AyahVerse at HalluScoring-2026: Category-Aware Arabic Hallucination Detection with Multi-Layer Representations
Abstract
AbstractWe present our submission to HalluScoring 2026, an ArabicNLP shared task on hallucination detection and answer recovery in Arabic LLM outputs. For Track 1, our submitted baseline fine-tunes CAMeLBERT for binary hallucination detection, achieving 0.921 and 0.763 AUC-ROC on the development and test sets of Task 1.1, and 0.749 and 0.729 on Task 1.2, respectively. We further conduct post-submission ablations with intermediate-layer supervision and category embeddings to study their effect on generalization. The combined model achieves 0.929 AUC-ROC on Task 1.1 and 0.777 on Task 1.2 development sets. For Track 2, we separate hallucination detection from answer selection, using CAMeLBERT for detection and a decoder-based, QLoRA-tuned AceGPT-v2-8B-Chat for option selection. The system achieves combined scores of 0.784 on Task 2.1 and 0.829 on Task 2.2 on the validation sets, improving substantially over the corresponding zero-shot baselines.