I've played with a square broach, and that follows the lead of the round pilot hole. The broach cuts equally (provided it enters straight) so the hole ends up on the same axis as the pilot hole.
But this "D" broach is a different animal. How do you determine that the circular segment of the finished hole is co-axial with the OD of the work piece? On the face of it the flat surface of the "D" doesn't cut, so the material removed is one sided, as shown by the swarf. That means that the centre of the "D" moves sideways as the broach enlarges the circular segment.
Or have I got it wrong, and it's actually very simple?