You are effectively plunging, which is prone to problems because the bit will want to “roll” around the inside of the hole you’re making. There is no good way to do this except for making the machine stiffer (usually not possible without rebuilding), making the hole larger, or avoiding plunge altogether with pre-drilled holes or a tool change to a drill bit.
A single flute cutter is less prone to rolling around the inside of a hole but by no means a guarantee. Also going as slow as possible might keep the cutting loads down but once it bites it tends to run away and you’re done.
I have the same spindle as you and I don’t have these problems. As Jamie suggested I use a single flute carbide end mill (have tried 3-6 mm) without any issues when plunging. Try this type of cutter and see if it helps.
I had this happen a couple days ago while cutting 3/4" plywood with a 1/8" two flute bit. I had cut dozens of parts prior without issue.
I discovered four issues while investigating. #1 was I had clamped down one side of the board but forgot to clamp the other side.
#2, not only did I forget to clamp the other side but the board was on top of the clamp causing that corner to be 1/2" above the sacrificial board in that corner. Major brain fart on my part.
After taking care of 1 and 2, I tried a test cut using a 1/4" router bit that I’ve used in the past with no issues. I didn’t get the vibration but I did notice that the bit was all over the place and moving too fast. This led me to discover issues 3 and 4.
#3, I discovered a couple mm of free play in the center gantry. This wasn’t there previously so I’m not sure if the vibration caused an existing issue to become apparent. One of the lock nuts was barely in the nylon and the bearing wasn’t touching. I thought the PLA had cracked but it looks fine. I snugged up the nut and all is nice and tight again.
#4, the feedrate for the tool I use somehow went back to defaults which is a bit too fast for the size of my mpcnc.
After fixing issues 3 and 4, I’ve had no more problems.
