Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Uncertainty Meets Conformance: A Process Mining-based Multi-Objective Evaluation for Clinical Reasoning using LLMs

AbstractWe present a framework for evaluating LLM-based clinical reasoning using conformance metrics from process mining. Four local LLMs were tested on EMR data from 16 vertigo patients across four prompting methods, with five repetitions per combination, yielding six uncertainty metrics. Reasoning traces were converted into event logs and analyzed with Heuristic Miner to derive external conformance to other patients’ reasoning patterns and internal conformance across repeated runs. Treating these as joint objectives, Pareto, Hypervolume, and TOPSIS analyses identified distinct evaluation scenarios, showing that conformance metrics can complement outcome-based uncertainty by revealing how LLMs reach their conclusions.

More from EMNLP 2026

All EMNLP 2026 papers →