Lexical Coupling in GUI Element Grounding: Sentence Embeddings Track Labels across Mobile and Web
Abstract
AbstractGUI grounding evaluations often treat high instruction–element embedding similarity as evidence of semantic grounding. Across three mobile and web benchmarks (AndroidControl, MoTIF, Mind2Web), we show this interpretation is frequently confounded by visible-label recovery: no off-the-shelf encoder significantly exceeds a BM25 lexical baseline at top-1, and encoder hit@1 is predictable from lexical rank, candidate-pool size, and label type at AUC 0.85–0.88. On AndroidControl, oracle fusion could lift R@1 by +12.8 pp, but standard reciprocal-rank fusion of the same signals instead degrades it by 1.4 pp. One epoch of in-domain contrastive fine-tuning of a small 22M encoder then beats every lexical and zero-shot baseline by +13.6 pp, localising the failure to off-the-shelf encoders rather than dense retrieval. These results suggest that, on its own, embedding similarity is not sufficient evidence of semantic GUI grounding beyond visible-label recovery; it should be reported against lexical controls and label-type stratification.