Holmes-Graph: A Framework for Probing the Linguistic Competence of Graph-Based Text Embeddings
Abstract
AbstractWe present Holmes-Graph, a framework to evaluate the linguistic competences of graph-based text embeddings generated by training graph neural networks (GNNs) on graph representations of text. The framework extends the Holmes benchmark, which previously covered only sparse, static, and transformer-based representations. The framework is flexible to be used with any graph representation and GNN architecture. Once weights are learned on a downstream task, frozen embeddings are probed across five linguistic competences. When applied to AI-generated text detection, Integrated Syntactic Graphs (ISG) combined with Graph Neural Networks (GNN) achieve an aggregate Holmes score of 0.662. This matches or exceeds DeBERTa-base on syntax, morphology, semantics, and reasoning, but not for discourse. The results confirm that graph-based text embeddings encode non-trivial linguistic knowledge and motivate their inclusion in future probing benchmarks.