Unibuc-NLP at SemEval-2026 Task 10: Unmasking Conspiracies with Pre-Trained Language Models

AbstractThe paper describes the system submitted to SemEval-2026 Task 10 (PsyCoMark) Subtask 2: detecting whether a Reddit comment expresses a conspiracy belief. We investigate three modeling paradigms: (A) an embedding-and-classify pipeline using Jina-embeddings-v3, HateBERT and BERT-Sentiment with Optuna-tuned classical ML models, optionally enriched by 19 readability features from textstat; (B) end-to-end fine-tuning of encoder transformers (DeBERTa-v3-base, DistilBERT) with a compact 128-unit classifier head and multiple pooling strategies; and (C) parameter-efficient QLoRA fine-tuning of large decoder-only models (Mistral-7B-v0.3, Qwen3-0.6B). Our best system, DeBERTa-v3-base with a 128-dimensional classifier, achieves a weighted F1 of 0.74, ranking 29/52 on the official leaderboard. Post-submission analysis further reveals that a weighted pooling strategy outperforms [CLS] on the official validation split by +0.04, achieving a weighted F1 of 0.78 (rank 8/52), suggesting that conspiracy-relevant features are distributed across transformer layers rather than concentrated at the final output.