ZYC at SemEval-2026 Task 5: Application of BERT-based Contextual Embeddings Similarity for WSD

AbstractWe investigate contextual embedding manipulation for Word Sense Disambiguation (WSD)as part of SemEval-2026 Task 5. We propose four approaches built on BERT-like pretrainedmodels, experimenting with the informativeness of similarity calculations and classificationmethods. We introduce scratch-trained cross-attention mechanisms inspired by GLiNER to compute similarity between definition or synonym representations and the full context. Our best performance achieved 57% accuracy with a Spearman correlation of 0.20. Our results suggest that finetuning strategy and trainng curriculum matter more than pretrained model choice for this novel task, and we identify several directions for future improvement. View our code base at: https://github.com/heliosraz/SemEval52026