The only good thing that i will say about cast iron, is that apart from the chilled skin, it is easy to machine, because of the graphite content, otherwise, it is PITA! The dust flies everywhere, and as you say everything gets dirty., and any sand is abrasive.
My method is to place a powerful magnet beneath the chuck and cover it with newspaper (tucked so the chuck will not catch it). When machining is finished, or when the pile of swarf becomes too large, the newspaper is carefully lifted, and used to carry the swarf to wherever it is going to be dumped.
In my case, this will be a clean tin that once contained peas, broad beans or baked beans. When sufficient has been put into the tin, the lid is replaced and hammered down, ready for placing in the recycling bin. The process is not absolutely clean, but much better than giving the swarf free rein to contaminate everything within a few feet!
Howard