Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Feedback Blindness: Diagnosing Silent Failures in Closed-Loop Agentic Data-Generation Pipelines

AbstractClosed-loop synthetic-data pipelines combine planning, LLM generation, verification, and deduplication; accepted outputs steer later generation. We study a production pipeline for Vietnamese contact-center compliance data and identify feedback blindness: an internal feedback mechanism stops serving its intended function while every stage completes normally, leaving only downstream metric degradation as evidence. We diagnose four instances: cumulative-count eviction empties a recency window; batch-level deduplication misses candidates scored before any batch member is committed; static boundary exemplars repeatedly anchor generation; and a fixed-size anti-repeat reference set loses informativeness as the dataset grows. We describe architecture-level fixes, mechanism-specific health signals, and a controlled ablation and scaling evaluation. The central lesson is that aggregate duplicate and acceptance rates are insufficient for diagnosis; pipelines should also monitor the feedback mechanisms that produce those outcomes.

More from EMNLP 2026

All EMNLP 2026 papers →