Nigel,
A couple of possibilities;
The end of a new piece of silver steel is often hard and either needs to be heated and slowly cooled or the first 10mm cut off with a hacksaw before machining. You might already know that because you say that your larger centre drills are fine?
I once bought a couple of duff small centre drills that wouldn't cut – so are yours ok? You could check by using one on something like aluminium or brass – just to check it cuts ok.
Yep, very often, small centre drills are run too slow and they struggle – so you could as you suggest look at the speed. Lubrication helps but a quick dip with a centre drill shouldn't need that much – i.e., a quick squirt of oil of coolant if you have it.
Phil H