Shared-Encoder Contrastive Grounding for Pose-Based Indian Sign Language Translation
Abstract
AbstractThe WSLP 2026 shared task studies sentence-level translation from Indian Sign Language pose sequences to English, with training on iSign and evaluation on a disjoint collection of source videos. We address two challenges in this setting: unstable pose scale caused by landmark errors and weak grounding between pose inputs and generated text. We introduce robust clip-level normalization and shared-encoder contrastive grounding, which aligns pose and target-text representations in a pretrained T5 encoder during training while retaining standard autoregressive decoding at inference. On a video-held-out iSign development split, contrastive grounding improves chrF from 0.102 to 0.121, while clip-level normalization further improves it to 0.128; the normalization gain is consistent across three matched seeds. A retrieval probe provides complementary evidence of improved pose–text alignment. Our final shared-task submission achieves chrF 0.16 on the official test set. However, input-independent English controls reach chrF 0.15, showing that small differences in the official score are difficult to interpret at the current level of translation accuracy. The source code is available at https://github.com/gyan-shrestha/WSLP-2026-SLT.