I would like a set of spot drills but can't justify the cost yet. I am no expert, and probably most of what I do is voodoo, but what I do to start the hole is;
If free hand drilling use a center punch first, on the mill i dont bother.
Then I use a center drill, but all i do is make a dimple, I dont go any further than the conical end in the tip. So really I use the center drill as a very stiff small spot drill.
Then with the drill take it gently to start so the drill can find the dimple. At this point just before the drill cuts I look for any side ways deflection and re align the work if needed.
After that a gentle feed, as I was taught, "Don't push it, let the drill (saw) do the cutting"
And of course the deeper the hole the more frequent backing out to clear swarf.
Finally as others have said, good and sharp drills. I have two sets, general making holes and a more expensive set reserved for the mill and special holes.
Adrian