Thanks (as usual) for all the feedback.
I've checked the cutter I've been using. This shaft measures around 0.01 mm ( 2 tenths) undersized (9.99mm) though it's possible that the flutes are a tiny bit smaller – not easy to measure.
It is a tight sliding fit in the collet which measures (best I can with calipers) pretty spot on 10mm.
Good news I thought. I cleaned both with denatured alcohol and mounted them in the quill.
I've got a new torque wrench but had to make suitable socket to go over the 12mm drawbar. I would have happily bought one but couldn't find such a thing.
The socket was constructed from some mild steel bar drilled one end at 9.5mm and milled to 12mm the other and then squared up with 2mm carbide endmills. I had several approaches to this each of which lost me one end mill. In the end I was able to take 2mm deep cuts going round the square and it worked. For various reasons the socket is a bit of a mess, so no photos!
I applied what I believe to be around 28Nm of force and proceeded to mill the same piece.
10mm deep. I started at 1 mm depth of cut, but got brave and ended up with 2mm depth of cut. I cut 3 stripes at 2mm (6mm in total) conventional milling and then 3 more 2mm stripes climb milling.
As best I can tell there is was no movement between cuts.

I'm not sure if you can tell from the photo, but there is almost no ridge to the finger touch on the top. Scraping a finger nail there is the slightest hesitation. In short this is about as good as it gets.
So it would seem that my earlier error was simply not to tighten the drawbar enough.
Having said that 28 Nm felt like a LOT at the end of the torque wrench and it was VERY hard to undo with the 6 inch spanner I normally use.
Is my general solution just to get a longer spanner? I'm not particularly keen on messing with a torque wrench to set tools.
Iain