A Study of Self-trained Unsupervised Semantic Segmentation Based on Dual-Branch

Semantic segmentation, a core field of computer vision, is essential for image understanding. Training such networks requires numerous fine-grained, pixel-level labels, whose acquisition is labor-intensive. Unsupervised semantic segmentation methods leverage labeled or easily labeled source datasets together with unlabeled target-domain images to achieve high accuracy on the target-domain test set, reducing annotation cost and becoming a research hotspot. However, after pre-training on large-scale datasets, existing unsupervised methods often extract insufficient semantic information when fine-tuned and directly applied to the target domain. To address this, we propose a dual-branch unsupervised semantic segmentation method. To mitigate issues in Transformer-based approaches, we add a semantic branch to the backbone to capture semantic context. Because the target domain lacks true labels, while retaining teacher-generated pseudo-labels, we introduce a dual-branch internal loss that uses student-generated pseudo-labels to guide the semantic branch, enhancing its ability to extract contextual information. In decoding, we improve feature fusion with a polar self-attention mechanism. We evaluate on two main unsupervised domain semantic segmentation tasks, GTA5 → Cityscapes and SYNTHIA → Cityscapes. Compared with the baseline, mIoU increases by 2.6% and 3.7%, respectively, significantly improving segmentation performance.