Drill bit diameter for Z axis Hole

I’ve been having trouble drilling the hole for the z axis properly.

Every time I do it, the hole don’t line up properly and I can’t assembly the darn thing.

What sizes hole have people drilling? I tried printing a spare piece to use as my guide and I even clamped it to the print and it didn’t work.

Will drilling a larger hole actually help or do I need to print a jig to clamp to the pipe for it to work with my press?

Is is the 3rd weekend I haven’t been able to over come this problem and would greatlly appreciate any and all help. Espicallyu since I was going to use the MP CNC as a 3d Printer but if I can’t assemble the first one the second one has no hope.

Thanks in advance for any and all help.
~John

The one for the tool mount? In the drawing I call for an oversized hole, up to 8mm if you have to. This hole does not need to locate anything it is a clamping hole.