Even if all the slack is taken up (etc) so the carriage etc cannot swivel, this can still happen. The cutting itself puts stress on the lathe parts, pushing them away from the workpiece. As soon as the stress is off, – ie you get to the end of the cut – the metal springs back to where it was before you started. Return the cut, and there is much less stress pushing the tool away, as most of the metal has been removed, so the distortion is less and so the tool is pushed away less. But it is still pushed away a bit. And it takes a tiny extra shaving off.
There is another effect causing this to happen. On the first pass, you are cutting a thread, in effect. Not a proper V thread, but a spiral of some sort, depending on the exact shape of the tool and the rate of feed. On the return pass, you follow the opposite hand of spiral, and so the tool can remove metal from the high ridges (even though they may be only 0.0005" deep) and so you notice extra swarf.
So – nothing to worry about, as really, in the real world, impossible to prevent.
Cheers, Tim