Thanks for the replies, I did actually manage to do it with a 3mm end mill, I had thought the metal would be too hard for the cutter to get a grip on but it worked very well as I took it really slow moving the cutter horizontally in the slot. It’s not exactly a superb example of milling but it did the job and the collet fits!
I did not get an email showing I had replies to my post for some reason