LLM-assisted Data Quality Improvement: SRL as a Case Study
Abstract
AbstractAlthough large language models (LLMs) perform much worse than traditional supervised models on the semantic role labeling (SRL) task, our closer investigation reveals two important findings. First, the existing labeled SRL data is substantially flawed, i.e., of unsatisfactory quality. Second, LLMs can serve as an effective tool for detecting annotation errors and deciding which labels require human review, from the perspective of minimizing human effort. This paper proposes an LLM-assisted data quality improvement approach, and applies it to two SRL evaluation datasets from two different domains. With our approach, human annotators need to review only 11% of the originally annotated arguments flagged for manual inspection, among which 40% are truly erroneous. Meanwhile, our approach suggests 20% new arguments potentially missed in the original labels for human review, among which 60% are indeed missing. In total, our approach locates and corrects 21% errors in the two evaluation datasets. Experiments show that both traditional supervised models and LLMs perform much better on the revised evaluation datasets.