Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Lexical Coupling in GUI Element Grounding: Sentence Embeddings Track Labels across Mobile and Web

AbstractGUI grounding evaluations often treat high instruction–element embedding similarity as evidence of semantic grounding. Across three mobile and web benchmarks (AndroidControl, MoTIF, Mind2Web), we show this interpretation is frequently confounded by visible-label recovery: no off-the-shelf encoder significantly exceeds a BM25 lexical baseline at top-1, and encoder hit@1 is predictable from lexical rank, candidate-pool size, and label type at AUC 0.85–0.88. On AndroidControl, oracle fusion could lift R@1 by +12.8 pp, but standard reciprocal-rank fusion of the same signals instead degrades it by 1.4 pp. One epoch of in-domain contrastive fine-tuning of a small 22M encoder then beats every lexical and zero-shot baseline by +13.6 pp, localising the failure to off-the-shelf encoders rather than dense retrieval. These results suggest that, on its own, embedding similarity is not sufficient evidence of semantic GUI grounding beyond visible-label recovery; it should be reported against lexical controls and label-type stratification.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Computer Science

More from EMNLP 2026

All EMNLP 2026 papers →