Enhancing Interactive Gaze Behavior Recognition via Co-training with Temporal Gaze Segmentation

Dynamic gaze recognition based on video sequences faces multiple challenges, including complex factors such as multi-person gaze tracking, gaze pattern transitions, and gaze point variations. We note that this task aligns closely with temporal gaze segmentation, which aims to identify the start and end boundaries of gaze behavior in videos. To this end, we propose a dual-task collaborative training framework with a temporal consistency loss. Here, gaze recognition and temporal segmentation are simultaneously optimized while explicitly constraining the confidence of gaze patterns according to temporal boundaries. In addition, to encode multi-factor contextual relationships, we employ an adaptive scene and gaze-heatmap interaction module. Through a cross-attention mechanism, it dynamically captures semantic correlations between gazes and scenes. Furthermore, to validate the method’s effectiveness, we establish a new gaze behavior dataset containing basic dual-person interaction scenarios, annotated with gaze pattern categories and temporal segmentation labels. Experimental results demonstrate that our method significantly improves the performance of gaze behavior recognition and exhibits superior generalization capabilities for handling multiple gaze patterns compared to other methods.