I don’t have a clue about this power feed device. I just see a pic of a shaft and a part which needs to fit together.
First, which way does that piece fit over the shaft?
How much do you need to enlarge that hole? Doesn't look like a lot.
Perhaps these people giving info on broaching do know of this particular power feed, but personally I would not want to be broaching through that length of material with that width of broach unless the broach was quite a ’deep’ one. I have the facility to press a broach, but have always found an alternative approach … so far.
Howard and Brian, above, offer sensible alternatives, one of which I would adopt. But even if using my hand shaper, I would relieve as much of the unnecessary section before starting to cut.