The broach goes from 15mm to 16.65mm in one pass. The shim is 1.65mm thick so this broach must be that cheapo, coarse cut jobbie you describe. The tooth form dictating the shim.
Curiously it has 22 teeth so I expected 0.075mm / tooth, but the chips are over 0.4mm thick and nowhere near the 60mm length of the cut. Part cut, part crunch it would seem.
Anyway, the keyways are cut, the job is done, the broach survived. I have some aluminium to cut next, that has to be easier. Doesn't it?
I can't see that slotting the arbour would cause significant bending. I put a 15x10mm trench in the broach guide and that seemed to stay straight. A longer 6x3mm groove is nothing. Is it?