TFB at SemEval-2026 Task 4: Diagnosing Model Failures in Narrative Understanding

AbstractWe describe the participation of team TFB in SemEval-2026 Task 4 on narrative similarity. We explore ColBERT-inspired sentence-level late interaction to capture event reordering, compare fine-tuning with synthetic data at multiple difficulty tiers, finding that distribution proximity to the target data matters more than volume and evaluate chain-of-thought prompting. We complement our approaches with a human annotation study (Krippendorff’s alpha=0.32) confirming the task’s inherent difficulty, an analysis of synthetic data distribution shift explaining why fine-tuning on out-of-distribution data hurts the model’s performance. Despite our tests, we didn’t surpass results of sentence-t5-xxl on Track B and Qwen2.5-7B on Track A. We finally decided to submit these two models for the task.