Does your rotary broached hole look to small? Sometimes a twist or spiral may result in the form from drag on the broach relief angle. Some solutions to correct this might be:
1. Redesign the broach.
2. Reverse spindle direction part-way through the hole.
3. Use an alignment fixture. (shown below)
Alignment fixture sketch
