Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Kinship Data Benchmark for Multi-hop Reasoning

AbstractMulti-hop kinship reasoning is a natural testbed for LLM compositionality, but existing benchmarks (notably CLUTRR) cover only the descriptive Eskimo system. We introduce KinshipQA, a procedurally-generated benchmark covering seven anthropologically-documented kinship systems (Eskimo, Sudanese, Hawaiian, Iroquois, Dravidian, Crow, Omaha) and up to six reasoning hops, with a tunable simulator horizon that eliminates exact-instance pretraining overlap. Evaluating six LLMs, we find a 40.9% accuracy drop when reasoning shifts from biological multi-hop to culturally-marked classification on the five non-descriptive systems. The drop follows the anthropological hierarchy descriptive > bifurcate > generational > skewing, persists under chain-of-thought and few-shot prompting, and compounds with depth (10.6% at 5–6 hops). Humans given the override rule reach 89.0% versus 50.7% for LLMs, placing the failure in the model rather than the task. Two follow-up experiments suggest distinct contributors. A fictional-rule control swapping system labels and kin terms for invented strings raises accuracy by 6.1%, implicating familiar English surface forms. An in-context-rule probe prepending the override rule helps skewing systems (+17.1%) but hurts non-skewing systems whose baseline already exceeds ~60% (-13.4%), consistent with a missing skewing prior alongside rule interference where the model already has a working approximation.

More from EMNLP 2026

All EMNLP 2026 papers →