Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Holmes-Graph: A Framework for Probing the Linguistic Competence of Graph-Based Text Embeddings

AbstractWe present Holmes-Graph, a framework to evaluate the linguistic competences of graph-based text embeddings generated by training graph neural networks (GNNs) on graph representations of text. The framework extends the Holmes benchmark, which previously covered only sparse, static, and transformer-based representations. The framework is flexible to be used with any graph representation and GNN architecture. Once weights are learned on a downstream task, frozen embeddings are probed across five linguistic competences. When applied to AI-generated text detection, Integrated Syntactic Graphs (ISG) combined with Graph Neural Networks (GNN) achieve an aggregate Holmes score of 0.662. This matches or exceeds DeBERTa-base on syntax, morphology, semantics, and reasoning, but not for discourse. The results confirm that graph-based text embeddings encode non-trivial linguistic knowledge and motivate their inclusion in future probing benchmarks.

More from EMNLP 2026

All EMNLP 2026 papers →