A few thoughts.
You mention marking out if this is a heavy scribed line and even more so if followed up by a dot & ctr punch mark then if that is not in exactly the same place as what is likely to be a more accurately located position from edge finding and use of handwheel & dro then if there is any play in the machine it will pull the drill over as it starts and its downhill from there. Either locate the punch mark on the mill by ctr finding or what I tend to do is just use the mill to get the location from edges the DRO makes this even easier, a simple sharpie mark is enough as a double check you have the hole in the right place.
As said above I just keep ctr drills for when I need the 60deg cone otherwise it's a dimple with a spot drill.
Good sharp drills help a lot, as you need to replace them get good ones at least in the common sizes ideally with split or four facet ends these seem to cut easier which means you are putting less force into the quill lever that may flex the machine. I don't bother to try and sharpen small ones.
On the smaller bench top machines I would tend not to drill larger than 6mm straight off anything larger use pilot first, not too small as thin drills can wander more. Again less force needed to drive a 12mm drill into say a 6mm then 9mm hole so less strain/flex on machine.
In the desire to get zero tram the column base is often shimmed which can work very well however the factory may have set the machine up for good all round use, you also need to take into account if the vertical Z movement is in line with your now shimmed column or perpendicular to the table. Like wise is the quill moving perpendicular to the table too. You may need to come to a good all round compromise as if one of these two is out then the further up the head goes to accomodate a long drill the further from the spot drilled position it will be in X&Y. Same with quill the further it gets extended the more it becomes out of line.
With the above in mind I'm also a fan of stub length drills, again in the common sizes and split point, they save having to risk errors by reducing head movement and they also save you having to keep cranking up the head to get long drills in.
Use your ER collet holder where you can as its probably truer than the average drill chuck and shorter too.
Make sure your work is tapped down flat in the vice particularly if using a drill vice as the moving jaw can lift so you are starting your drill on a slope which it will want to wander down and the hole won't be perpendicular to the surface so it may actually be straight but as it was drilled at an angle will seem wonkey