Whenever I use a slot drill to machine a fork end or slot I usually end up with an oversize irregular shaped slot.
You have slop somewhere and the wonky slot proves it. Stiffness is everything when you mill in a lathe.
If you are making a fork you can use the side of the cutter instead of the tip btw, don't use too high a speed, a nice whirring speed(300-500) that munches the metal away quietly works best for me
Usually the milling slide is the weakest point, I had to beef up my ML7 one on my M series Drummond with a better chunkier cross slide t-bolt and I also use stops to prevent it from twisting under load
Then there's the other slides, they all need nipped up
If you get any noise instead of a nice quiet whirring munching sound then something ain't right