Benchmarking Legal AI on Ambiguity: A New Evaluation Framework for Edge-Case Reasoning

Abstract

Existing legal AI benchmarks are largely limited to deterministic accuracy measures. However, legal practice involves many more tasks in the presence of uncertain statutes, conflicting case law, and interpretive disagreements. Existing benchmarks do not evaluate these tasks in circumstances where there is no clear consensus and settled point of law. The article presents an Ambiguity-Aware Evaluation Framework that distinguishes three characteristics of legal questions, like deterministic questions with clear legal answers, contestable questions with legal uncertainty due to reasonable interpretive disagreement,  and indeterminate questions requiring policy value judgments or fact-sensitivity. The article introduces novel metrics, such as ambiguity detection rate, interpretation completeness, and qualification appropriateness, which allow for rewarding systems that recognize and handle legal uncertainty. Architectures for ambiguity detection, systematic enumeration of interpretations, and uncertainty calibration can help yield more practically acceptable outputs from legal AI systems, as they avoid overconfidence in contexts that require epistemic humility, such as when legal rules are indeterminate and entail multiple valid interpretations.