Beyond Dense Retrieval: A Structured-Semantic Approach for Conversation Analytics
Abstract
AbstractLarge Language Model (LLM) based conversational systems generate vast interaction logs containing critical diagnostic signals such as failure modes and safety violations. However, the cross-turn dependencies, implicit state, and topic drift inherent to these conversations defeat dense retrieval, whose lone embedding cannot concurrently encode discrete structured constraints (e.g., refusal type) and open-ended semantic content. Our central claim is conceptual: conversational retrieval must span two complementary information planes, structured metadata and semantic content, and these are best reconciled by a unified hybrid representation that jointly supports filtering over structured metadata fields and vector similarity search over unbounded open-ended fields. We instantiate this as the Extract-Annotate-Query (EAQ) framework, which derives a metadata taxonomy, applies LLM-driven annotation, and compiles natural-language queries into hybrid graph operations. On the CDR-Benchmark, EAQ lifts NDCG@10 by 30.8% over the strongest dense baseline, with consistent gains across five analytical categories and two external datasets.