Does your rotary broached hole look to small? Sometimes a twist or spiral may result in the form from drag on the broach relief angle. Some solutions to correct this might be:

1. Redesign the broach.

2. Reverse spindle direction part-way through the hole.

3. Use an alignment fixture. (shown below)

Alignment fixture sketch