User-Level Dynamic Beam Hopping Design for LEO Satellite Networks Based on Deep Reinforcement Learning Assisted Enhanced Genetic Algorithm

Beam Hopping (BH) is a promising approach to support the dynamically varied and non-uniformly distributed ground traffic demands with limited satellite beam resources. However, almost all the existing BH schemes only focus on the overall cell-level traffic demands without considering the transmission demand of each user, which may degrade the system performance. To address this issue, in this paper, a user-level dynamic BH scheme for low Earth orbit satellite networks is proposed, where the user-level real-time traffic demands are integrated into the BH pattern design. Specifically, by considering the user-level transmission demands in the multi-satellite and multi-cell scenario, we formulate an optimization problem that aims to maximize the overall long-term throughput of the network by jointly optimizing the BH pattern and access control (AC) strategy. To solve the formulated problem, we first establish a user-oriented Markov decision process framework, based on which the original long-term optimization problem can be converted to a short-term sum value maximization problem. Then, a deep reinforcement learning assisted enhanced genetic algorithm is proposed to solve the converted short-term optimization problem, where the deep reinforcement learning is adopted to estimate the long-term state-action values and the enhanced genetic algorithm is used to determine the BH pattern and AC strategy with a low complexity according to the estimated state-action values. Simulation results show that the proposed scheme can achieve better performance over existing methods.