James, even with a parallel keyway when you start the cut only the first tooth is in contact with the egde of the hole, the “lead” of the broach is not in contact so there is just as much uneven contact doing a straight cut as a taper.
It is only as the first tooth reaches the end of the cut that you have full contact which on our sort of size flywheels and keys may be 6-10 teeth and that is when the force needed to push the broach through is at its highest.
Sometimes I borrow a friends hydralic press which is really needed for the 3/16″ broach but below that I can get away with teh drill press but for 1/8″ I eill push the 3/32″ through first to make it easier on the drill and also my right arm. I put a block under the table on the drill as you can see a bit of flex otherwise.