Ah … I see your point, Mark
Sorry, I can't remember what taper is on the Broaches, but I believe it's standard.
You will [hopefully] find that the Broach is a shallower taper than the pin and, as you say, hammering in the pin will spread the hole slightly.
Don't forget to support the Plate on a suitable Staking Block.
MichaelG.
P.S. The old Clockmakers would just turn the taper onto a straight pin.