I have notices when I use a mill to drill a partial hole and then chage to a drill the drill is off ctr and vica vera.
This photos shows a puch mark done with the drill chuck and then the mill chuck and as you can see the mark is in the same position.
However I then drilled a 6mm hole, remove the chuck and fitted the collet chuck and as you can see in the second photo there is a step. The table has not been moved
Are you saying you are re-drilling the same hole, not starting a new one at a new centre punch position. If you a trying to use a two flute drill in an already partially drilled hole then put the drill into the hole before running the spindle. This will stop the drill from snatching in the hole and pulling of centre. There was a discussion recently regarding trying to enlarge an already drilled hole, in that case you need to use a drill with more than two flutes. Using more flutes centres the drill on the already performed hole.
As Tony says it sounds like a tram issue, as the head/quill will be nearer the work when using a short milling cutter rather than a longer drill the error will show the result you are getting
Perhaps the rough button thingy isn't being held firmly on the mill's table. You get odd shaped holes if work moves whilst it's being cut. (You can guess how I know…)
An easy way to check the tram is with a DTI at right angles to the chuck as shown in the picture.
By swinging the DTI round in a circle on the table top, you can measure how far the head and table are out, right/left and forward/back. The DTI being on an arm means you can see tilt of 0.01mm over a distance of 200mm quite easily.