IWC-DiF: Learning a generalizable NeRF by incremental weight consolidation of dictionary fields

Training a generalizable NeRF across multi-object is a challenging task. Existing mixed training strategy (e.g., Dictionary Fields (DiF)) is impractical when the amount of objects is largely increased. To solve this issue, we treat it from the perspective of incremental learning and propose a novel method to learn a generalizable NeRF by incremental weight consolidation of DiF, namely IWC-DiF. In particular, it consists of an incremental training stage and a testing stage. In the training stage, the DiF (including the basis field and coefficient field) of the first task (i.e., object) is initialized and trained by the rendering loss of dense multi-view images. For each following task, its basis field is firstly initialized by the latest task and then trained by the rendering loss of current dense multi-view images and the uncertainty-weighted regularization loss of Elastic Weight Consolidation (i.e., the uncertainty-weighted Fisher Information Matrix of the basis fields of all past tasks). In the testing stage, the DiF of the new task is firstly initialized by the final trained task and then its coefficient field is fine-tuned by the rendering loss of sparse few-view images. Experiments conducted on the GSO dataset have revealed the detrimental catastrophic forgetting phenomenon associated with the original DiF method. Additionally, extensive experiments performed on both the GSO dataset and the CO3D dataset have demonstrated the remarkable generalization capability of IWC-DiF in both out-of-class and in-class multi-object tests, surpassing the performance of most state-of-the-art methods.