Joint State Space Enhanced DQN for Beam Tracking in Dynamic MIMO System

To address the critical challenge of beam tracking in millimeter-wave (mmWave) massive MIMO systems under highly dynamic mobility scenarios, this paper proposes an improved Deep Q-Network (DQN) algorithm based on a joint signal strength and angle dynamics state space. Conventional beam tracking methods, such as the Extended Kalman Filter (EKF), suffer from model mismatch and performance degradation under abrupt trajectory changes, while existing DQN approaches lack awareness of angular variation trends, leading to delayed response and increased misalignment risk. The proposed method enhances the DQN state space by incorporating both in-phase/quadrature components of received signals and the rate of angle change, enabling more accurate motion trend perception and proactive beam adjustment. A customized reward mechanism and discrete action space are designed to align with beamforming codebook constraints and channel dynamics. Simulation results demonstrate that the proposed DQN algorithm achieves robust beam tracking under varying initial errors and motion patterns, significantly outperforming EKF in terms of tracking accuracy, convergence stability, and adaptability to sudden trajectory changes, thereby offering a model-free and effective solution for beam management in 5G-Advanced and 6G mmWave systems.