I always engage the collet into the nut, before fitting to the arbor. (How else can you do it? )
The collet should be selected to be as close a fit on the cutter as possible.
And as asked, are the cutters sharp?
Remember that if the collet is not tightened sufficiently, the helix on the flutes will try to screw the cutter out of the collet.
Once I had learned to tighten the clamping nut fully, (The ball bearing type help ) I had no more problems with End Mills moving.
Howard