Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Beyond Caption Loss: Generative Distillation for Modality Alignment in Vision-Language Models

AbstractVision-language models align frozen vision encoders to language models using only a next-token prediction loss, an asymmetric signal that discards rich visual signals not captured in the captions. We propose GenDA (Generative Distillation Alignment), a novel training framework that augments the captioning loss with a generative distillation objective from a frozen text-to-image diffusion model. GenDA concatenates selected visual tokens with the encoded caption to form a joint conditioning signal, encouraging these tokens to encode visual information not already captured by the caption. Across five vision encoders and four language models, GenDA gains 3-9 CIDEr points over LLaVA-1.5 on five captioning benchmarks. These gains also transfer to downstream VLM benchmarks while preserving the encoders’ discriminative performance. Visual preservation and modality-gap analyses suggest that visual information is retained deeper within the LLM, while visual features become more closely aligned with the language representation space.

More from EMNLP 2026

All EMNLP 2026 papers →