In ultra-deep well drilling, the drill bit undergoes severe axial-torsional-lateral coupled vibrations, such as stick-slip and whirling, significantly increasing drilling risks. The evolution laws of these complex vibrations have not been fully revealed, making it challenging to design effective vibration mitigation strategies. Therefore, this paper aims to build a new drill bit dynamic model considering axial and lateral cutting processes to reveal the evolution laws of stick-slip and lateral vibrations. Firstly, by considering the three-dimensional cutting force while breaking rock, cutting limits, and nonlinear friction, the axial-lateral-torsional coupled vibration model of the drill bit was established. Next, the contact determination conditions between the drill bit and rock were extended, refining the three-dimensional bit-rock interaction model. Additionally, considering the complex time delay caused by axial-lateral-torsional vibrations, the realistic cutting profile for the drill bit was constructed. Consequently, the evaluation indexes for the stick-slip vibration and whirling of drill bit were developed. Finally, the validity of the model was confirmed through field data validation and comparison with the RGD model. The results indicate that the lateral motion of the drill bit undergoes a “collision-cutting-separation” process. The lateral alternating cutting motion intensifies stick-slip vibrations, reduces torsional amplitude stability, and raises the rotational speed threshold for escaping stick-slip vibration. As rotational speed increases, stick-slip vibration transitions to intermittent stick-slip vibration and eventually escapes from stick-slip vibration. The lateral cutting limit and torsional sticking state can lead to the lateral sticking state. With increasing rotational speed, the lateral motion changes from “collision-cutting-sticking-separation” to “collision-cutting-separation”, with the lateral motion evolving from random backward motion to quasi-periodic backward motion and ultimately entering periodic backward whirling. This study provides a theoretical foundation for drilling parameter optimization and drill bit selection.