Information-Theoretic Gating for Online Continual Mixture-of-Experts on Streaming Multimedia Data

Modern multimedia applications increasingly rely on large-scale models operating in dynamic, streaming environments, making robust and adaptive routing mechanisms essential. The gating mechanism is pivotal for Mixture-of-Experts in online continual learning, yet existing designs are largely heuristic. To bridge this gap, under the online continual learning setting, we analyze the role of gating strategies in shaping the Minimum Excess Risk (MER), which quantifies the discrepancy between oracle and learned expert performance. We theoretically prove that minimizing MER is equivalent to maximizing the mutual information between expert assignments and labels/outputs, which enables MER optimization to be performed using low-dimensional tensor computations. Building on this theoretical foundation, we design two novel loss functions grounded in mutual information, applicable to both fully labeled and label-free scenarios. To further guarantee computational efficiency, we develop a lightweight, matrix-based mutual information estimator with a rigorous joint entropy formulation, achieving a reduction in computational complexity from the conventional O(n³) to O(n²) computational complexity. Through extensive evaluations on MNIST, Fashion-MNIST, KMNIST, and EMNIST, our approach consistently outperforms SOTA baselines, reducing overall error by up to 12.3% and forgetting by 3.9%, both with statistical significance.