Privacy-Utility Trade-Off in Federated LLM Fine-Tuning: A Dynamic Game Approach

Fine-tuning large language models (LLMs) is critical for adapting pretrained models to specialized downstream tasks. Federated LLM fine-tuning enables privacy-aware model updates by allowing data owners (DOs) to contribute a global LLM without exposing local data. However, full-parameter fine-tuning in federated settings incurs significant computational and communication overhead, while frequent gradient exchanges increase the risk of privacy leakage, such as memorized data inference. Parameter-efficient fine-tuning (PEFT) with differential privacy (DP) offers a low-overhead alternative with formal privacy guarantees, but fails to strike privacy-utility tradeoff under heterogeneous privacy preferences: individual DOs may inject excessive DP noise to maximize privacy, whereas the curator aims to minimize noise to preserve model quality. In this paper, we present an innovative game-theoretical framework that enables dynamic privacy trading within differentially private federated LLM fine-tuning. In the game, DOs strategically adjust their local DP noise levels in exchange for customized incentives from the curator, thereby balancing privacy and utility. We begin by establishing a theoretical convergence bound that quantifies the influence of locally injected noise on the global model utility. Under this bound, we analytically characterize the pure-strategy Nash equilibrium of the game, accounting for DO heterogeneity, curator budget constraints, and noise estimation errors. For mixed-strategy settings with incomplete information, we design a hierarchical reinforcement learning algorithm that jointly learns DOs’ optimal noise-saving strategies and the curator’s optimal pricing policy without presupposing their private information. Experiments on real-world datasets demonstrate that the proposed scheme improves DO utility, reduces curator cost, mitigates free-riding, and accelerates convergence compared to existing methods.