跳到主要导航 跳到搜索 跳到主要内容

CheersA large glass mug filled with dark beer topped with a thick, foamy head. The mug has a handle and condensation on the outside, suggesting the beer is cold. The background is plain white, emphasizing the rich color and texture of the beer and foam. The image highlights the refreshing and appealing qualities of a freshly poured dark beer. Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation

  • Yichen Zhang
  • , Da Peng
  • , Zonghao Guo
  • , Zijian Zhang
  • , Xuesong Yang
  • , Tong Sun
  • , Shichu Sun
  • , Yidan Zhang
  • , Yanghao Li
  • , Haiyan Zhao
  • , Wang Xu
  • , Qi Shi
  • , Yangang Sun
  • , Chi Chen
  • , Shuo Wang
  • , Yukun Yan
  • , Xu Han
  • , Qiang Ma
  • , Wei Ke
  • , Liang Wang
  • Zhiyuan Liu, Maosong Sun
  • Tsinghua University
  • Xi'an Jiaotong University
  • University of Chinese Academy of Sciences

科研成果: 书/报告/会议事项章节 › 会议稿件 › 同行评审

摘要

A recent cutting-edge topic in multimodal modeling is to unify visual comprehension and generation within a single model. However, the two tasks demand mismatched decoding regimes and visual representations, making it non-trivial to jointly optimize within a shared feature space. In this work, we present Cheers, a unified multimodal model that decouples patch-level details from semantic representations, thereby stabilizing semantics for multimodal understanding and improving fidelity for image generation via gated detail residuals. Cheers includes three key components: (i) a unified vision tokenizer that encodes and compresses image latent states into semantic tokens for efficient LLM conditioning, (ii) an LLM-based Transformer that unifies autoregressive decoding for text generation and diffusion decoding for image generation, and (iii) a cascaded flow matching head that decodes visual semantics first and then injects semantically gated detail residuals from the vision tokenizer to refine high-frequency content. Experiments on popular benchmarks demonstrate that Cheers matches or surpasses advanced UMMs in both visual understanding and generation. Notably, Cheers outperforms the Tar-1.5B on the popular benchmarks GenEval and MMBench, while requiring only 20% of the training cost, indicating effective and efficient (i.e., 4× token compression) unified multimodal modeling. Code and checkpoints are available at github.com/AI9Stars/CHEERS.

源语言英语
主期刊名Computer Vision – ECCV 2026 - 19th European Conference, Malmö, Sweden, September 8–12, 2026, Proceedings
编辑Paolo Favaro, Zuzana Kukelova, Atsuto Maki, Anna Rohrbach, Konrad Schindler, Federico Tombari
出版商Springer Science and Business Media Deutschland GmbH
页398-417
页数20
ISBN(印刷版)9783032370976
DOI
出版状态已出版 - 2026
活动19th European Conference on Computer Vision, ECCV 2026 - Malmö, 瑞典
期限: 8 9月 2026 → 12 9月 2026

丛书

姓名Lecture Notes in Computer Science
卷17062 LNCS
ISSN(印刷版)0302-9743
ISSN(电子版)1611-3349

会议

会议19th European Conference on Computer Vision, ECCV 2026
国家/地区瑞典
市Malmö
时期8/09/26 → 12/09/26

学术指纹

探究 'CheersA large glass mug filled with dark beer topped with a thick, foamy head. The mug has a handle and condensation on the outside, suggesting the beer is cold. The background is plain white, emphasizing the rich color and texture of the beer and foam. The image highlights the refreshing and appealing qualities of a freshly poured dark beer. Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation' 的科研主题。它们共同构成独一无二的学术指纹。

引用此