Do bear in mind I've never received much formal training on this sort of thing.
I suspect the slot drill wasn't held either rigidly enough or central enough.

Last time I did something similar, I held an undersized slot drill in a boring bar holder in the toolpost.
Make sure it is central to the axis of rotation and also with the front cutting edge at centre height, just like a normal cutting tool.
Wind the slot drill into the workpiece, just like you would from the tailstock, and then outwards towards you until the hole is the correct diameter.
A carriage stop might be useful to get the depth spot on when re-cutting after measuring the diameter.

Bill

As Peter says going straight in with the final size cutter can see it cutting oversize, I would drill a pilot hole say 5mm, then use a 5mm milling cutter followed by the 6mm but would not expect it to be as accurate as say a reamed hole. For cutters I find the 3flute ones plunge better than 2 flute.

The other option is to plunge most of it with a milling cutter of 5mm or 7/32" after pilot drilling and then use a small boring bar to finish, as shown in the recent 14.5mm hole thread a milling cutter can be use das a small boring bar as shown here doing a tapered hole in a pulley (note setup for reverse spindle direction)

I would bore it with a 5mm slot cutter, used as a boring bar and held in the toolpost as above. 5mm pilot drill and then bore to size / length with the slot cutter. Small V block under the cutter if needed.

I'd start by checking the alignment of the tailstock.