Skip to content
conftrace_
2026 ICML ICML 2026

Position: Mathematical DSL Benchmarks Need Executable Interface Contracts

This position paper argues that documentation is infrastructure for reproducible geometry reasoning, and more broadly that mathematical benchmarks built around formal domain-specific languages (DSLs) need an executable interface contract: documented vocabulary should be matched by executable, versioned behavior and runnable examples. We use JGEX (as implemented by Newclid and AlphaGeometry) as a prototypical case study of how documentation–implementation gaps, missing examples, and nontrivial encoding choices can constrain expressivity, affect solver behavior, fragment tool interoperability, and bias benchmark construction. We introduce JGEXDiv, a curated collection of $137$ Euclidean geometry problems with released JGEX formalizations, natural-language rewrites, and metadata, covering $51.4%$ of admissible construction-pattern bins under our signature-based analysis. We also release a conformance suite for the documented JGEX interface, including minimal runnable examples, validation scripts, and a predicate-level support matrix. This audit surfaces cross-tool discrepancies, including documented predicates that execute under pinned Newclid v3.0.1 but are not recognized by Yuclid, as well as predicates that are unsupported or unstable under Newclid itself. Finally, we provide an executable NL$\rightarrow$JGEX baseline under the same backend. We argue that these artifacts together form an executable interface contract for mathematical DSL benchmarks, and should be treated as first-class research artifacts alongside datasets and evaluation code.

More from ICML 2026

All ICML 2026 papers →