I think old mart's suggestion of paying attention to the null point has perhaps not been given enough attention.
The terminology he uses might not be standard but the geometric concept behind the words is important. In a way it is similar to how setting a lathe tool above, on or below centre can have serious effects on its cutting ability.
Slater Tools, who know quite a lot about this process, stress the need to set up the tool before starting:
http://www.slatertools.com/support/internal-rotary-broach-toolholder-setup-instructions
Their broaching catalogue, here:
http://www.slatertools.com/resources/uploads/PDFs/ST-catalog-english.pdf
shows some interesting things and highlights that the length of the broach is important.
If you mix and match broach and holder as the OP has done, there seems to be considerable scope for poor cutting. With a tool that is too long or too short, the vertex they show in their diagram will not coincide with the cutting edge.