I can't readily imagine another cause than the 6mm slotdrill running out. If another 6mm cutter produces the same result, the most obvious reason would be swarf or grit in the collet cone, tipping its axis.
There may well be enough parallel shank exposed to get a clock finger onto it to verify concentricity, though 0,3mm TIR should be pretty clearly visible when running.
When I make salt/pepper grinders, I regularly mill a slot in the delrin cap with a 1/4" two-lip slotdrill held in a 3-jaw chuck in my Warco, with the the cap held in the vice on my vertical slide. This is always a tight hand-push fit for a piece of 1/4" square-section brass bar (usually measuring .249" ) .
So no, an oversized cut is not necessarily to be expected.
Hopper's point about starting with an undersized hole is good. There might be risk of deflection when an outer corner of the slotdrill contacts the work first, especially if the feed's severe. If the slotdrill runs true, best keep down the work it has to do.
Edited By Mick B1 on 26/11/2020 14:27:47