Improving modality alignment via multi-scale and selective feature learning in reasoning segmentation

Reasoning segmentation is an emerging and challenging task that requires models to accurately understand implicit queries and precisely localize visual targets for segmentation. Existing reasoning segmentation models typically employ visual instruction tuning to achieve cross-modal alignment. However, when confronted with complex inputs, this alignment approach, which solely relies on visual instruction tuning, easily leads to incorrect cross-modal associations. This issue impairs the model's ability to accurately capture complex semantic relationships, thus affecting the accuracy of segmentation and text responses. In this work, we present the Reasoning Segmentation Assistant (RSAT), an MLLM that enhances accurate interpretation of input information through cross-modal semantic interaction and multi-scale feature learning. Specifically, RSAT introduces an Attribute Feature Extractor (AFE), which extracts attribute information from query texts to guide the LLM to focus on key features and transforms them into high-level semantic features. Subsequently, the Modality Interaction Module (MIM) establishes precise semantic correspondences via fine-grained cross-modal interaction and multi-scale semantic fusion, substantially enhancing cross-modal alignment. AFE and MIM can be conveniently deployed in other reasoning segmentation models. Comprehensive experiments demonstrate that RSAT achieves improved performance across multiple reasoning segmentation benchmarks and exhibits strong competitiveness in referring segmentation tasks.