Enhancing scene graph generation via semantic-aligned masked vision-and-language pre-training

Scene graph generation (SGG) is pivotal for enhancing visual understanding, yet it faces challenges such as expensive training annotations and long-tail distribution issues. Recent studies attempt to address these challenges by using a fixed language parser and an object detector to obtain triplet labels for augmenting SGG data. However, this approach reduces diverse, semantically rich captions to structured labels. In this paper, we propose a novel masked Vision-and-Language (V&L) pre-training framework to improve SGG representation learning from free-form captions and image patches. We combine semantic-aligned masking strategies with multimodal masked modeling objectives to learn scene-related visual and textual representations. This framework can be further fine-tuned on SG data to achieve more accurate SGG results. Evaluations on the Visual Genome benchmark show competitive results in both the Language-Supervised and Fine-Tuned settings. Our framework raises Recall@100 from 4.1 to 8.5 under the Language-Supervised setting and outperforms most two-stage SGG models under the Fine-Tuned setting, highlighting the effectiveness of self-supervised pre-training on large-scale, noisy captioning data.