The test bar, or the mill spindle, tapers may be slightly wrong which is a reason why it could “wobble”. If it’s not visibly loose try coating with blue, or just some very soft pencil lines down the length of the arbor, then rotating it in the socket and checking that the “wear” on the marking is even.
I bought a large Chinese keyless chuck with a separate MT2 arbor and was puzzled as to why the chuck keeps rotating in the tailstock when drilling. Then I noticed that the arbor wobbled in the socket, its taper is wrong! At least I assume it’s the arbor, as the tailstock is by Myford and other MT2 arbors are fine in it.