Hello Mark, good to see you are still at it..
Partly by way of a ‘bump’ back up the list, with apologies for being a bit general, but there are others much better qualified than me to comment on the specifics of this particular job..
I think with ‘grabby’ material of this kind it sometimes pays to stone the cutting edges of the drill back a bit, so that they are less acute / aggressive – er, in the manner of lathe tools, make the ‘rake angle’ less positive and more ‘neutral’, if you see what I mean..
..perhaps also avoid using small steps when drilling, use a small pilot and then go straight to size – which I guess could be ‘reaming/boring size’ here?
I’d have thought that if you have capacity on the lathe, bolting the soleplate onto the cross-slide table might give a better ‘bed’ for operating on, more rigid and with better control of feed?