The grip length of an ER40 collet is too long really for holding 4mm x 25mm accurately. It's the main problem with this style of collet.
If I had a lot to do I suspect I would make something to hold them first. That could simply be a recess in something that was a snug fit using grub screw to hold them in place but you may have to avoid marking them so the screws would need some thought. Soft solder ends maybe. Or some steel plate on a surface plate. Drill a hole through it, slit into the hole from one edge, drill and tap a hole through the edge to clamp it up. Mount on the lathe face plate with the clamp screw tightened a touch and then machine the recess for the screw heads.
Soft jaws are another option and maybe normal a normal 3 jaw chuck with the jaws reversed. That way you will have a square on face to press the head up against. If you rotate the work by hand just as the jaws close on it things generally will run as accurately as the chuck does. There is a bit of a knack to doing that though. A small chuck will do this sort of thing easily. Big ones too if the part is big enough. It may be feasible to machine up a split ring if that's the case. You may find that you can't get the part out if you do that though.
Rotagrip stock soft jaws but be prepared to spend a while fitting them to the chuck by hand with a file. I think they are deliberately made a touch over size. Getting them to a really good fit usually takes me 1/2 hr or so being very careful to find out what aspect needs relieving.
They only need thou's removing as well.
John
–