I've noticed this effect on steel as well, and put it down to resonances though my machine-tools are not on tip-top condition or adjustment, and that won't help.
However I have also noticed it on lengthways turning, and that may well be due to resonance nodes, though I suspect also the saddle "walking" slightly.
'
The material fault Martin suggests is possible in some metals at least.
There are two curious, small shiny patches in the faced surface of a 6-inch diameter mild-steel flywheel I have just machined on a moderately hefty lathe with powered cross-feed. I could not see them during the cutting, but could hear the tool, a 6mm button, passing through them. As I was modifying a part I had made several years ago, I can't recall if the stock was a billet from round bar, or trepanned from plate.
When I was the materials store-keeper for a printing-machine manufacturer I once had complaints my issuing about a batch of allegedly sub-standard aluminium-alloy bar. The milling and anodising revealed a peculiar elliptical "core" right through the bar's length, as in seaside rock. They had to accept what I pointed out: I could have had no idea the interior of the metal was like that, it was from our usual supplier, and my hacksawing machine left rough ends that hid any defects.