Depending how long the part is, an ER40 may not be the best collet to use. 5C or similar will be better for short pieces, like less than 1 diameter long.
There are other collet types that are better holding as well, ones that have a shallower taper in the body, like the OZ collets or the BIG Daishowa collets.
Then if you need to be using an ER40 collet, buy a good quality ER40 collet in Ø6 mm or 1/4 inch, then set it in place with pin to ensure that it is running true. Not very tight in the holder, using an undersized pin, like 5.97mm for example. After taking the pin out, use some thin super glue to hold it all in place. Then either grind or turn it out to finished size with a sharp boring bar. Iscar make some really good bore bars for small holes, like 6mm for example. It will bore 36mm deep which is longer than what most ER40 6mm collets are with their small hole length.
Then soak in Acetone to dissolve the super glue or boil it all with soapy water will release the super glue. Mark or ID the radial position that the collet was in, and it will be as accurate as you can have on your setup.
When holding short pieces in ER40 collet at work, I have a support plug in the back of the collet at the same diameter as the part being held. It prevents the back from collapsing, and then allows the front section with the 30 deg taper to collapse a little bit to hold the part well. I find it rare that an ER40 collet will hold a part dead true from the initial closing. Usually it will to be lightly tapped to get it to be very true. Often it is within 0.02mm but not 0.005mm true.
The Big collets will hold a nominal diameter on initial clamping better than 0.005mm TIR right off the bat. Often even better if the spindle is any good.
Neil