FMW-Net: a first-order meta-weight-net approach for sample weighting

Deep neural networks (DNNs) have achieved impressive performance in various applications, but are susceptible to overfitting biases in training data, such as label noise and class imbalance. Example reweighting methods can be used to solve this issue, while often require manually specifying the weighting function forms. Recently, Meta-Weight-Net (MW-Net) method has been proposed to automatically learn the weighting function parameterized by a Multi-Layer Perceptron (MLP) in a meta-learning manner. However, the update of MW-Net suffers from expensive computations due to the second-order gradient computation in bilevel optimization. To address this issue, we propose a First-order MW-Net (FMW-Net) algorithm based on value-function approach, which relies solely on first-order gradient information. The novel learning algorithm has better scalability due to its lower compute/memory costs (compared to MW-Net, the time cost is reduced to approximately 33%, and the memory cost is reduced to 75%), making it both practical and efficient for large-scale models in deep learning, e.g., large language models. We present empirical results demonstrating its superior practical efficiency. Source code is available at https://github.com/ybzhouni/FMW-Net.