Legal Research Fidelity

Updated 7/1/2026

Industry Partner

Evaluates source-grounded legal research, citation validity, jurisdiction awareness, and calibrated abstention.

View:Task:


1Claude Fable 5 Run settings: temperature 0, max tokens 16384, extended reasoning, 5 repeated runs.	73.32% ± 0.82	$4.39	734s
2Claude Opus 4.8 Run settings: temperature 0, max tokens 16384, extended reasoning, 3 repeated runs.	70.10% ± 1.16	$3.18	1410s
3Claude Sonnet 5 Run settings: temperature 0, max tokens 16384, extended reasoning, 4 repeated runs.	66.41% ± 1.33	$2.80	662s
4GPT-5.6 Sol Run settings: temperature 0, max tokens 16384, extended reasoning, 6 repeated runs.	66.11% ± 0.99	$5.30	793s
5GPT-5.5 Run settings: temperature 0, max tokens 16384, standard reasoning, 6 repeated runs.	61.20% ± 1.67	$3.99	723s
6Muse Spark 1.1OPEN Run settings: temperature 0.2, max tokens 16384, extended reasoning, 5 repeated runs.	61.20% ± 1.50	$0.26	412s
7GLM 5.2OPEN Run settings: temperature 0.2, max tokens 16384, standard reasoning, 4 repeated runs.	59.19% ± 0.65	$0.56	543s
8Grok 4.5 Run settings: temperature 0, max tokens 16384, standard reasoning, 3 repeated runs.	59.01% ± 0.48	$1.64	412s
9Gemini 3.5 Flash Run settings: temperature 0, max tokens 16384, standard reasoning, 5 repeated runs.	57.36% ± 0.82	$0.61	256s
10Qwen 3.7 PlusOPEN Run settings: temperature 0.2, max tokens 16384, standard reasoning, 6 repeated runs.	56.92% ± 0.99	$0.48	444s

Motivation

Generic leaderboards rarely show whether a model can complete the work people actually depend on. This evaluation turns a real workflow into a controlled, repeatable test with visible uncertainty, cost, and latency.

Results

Industry average accuracy comparison

Mean score with standard-error whiskers across repeated runs.

Claude Fable 573.32% ± 0.82

Claude Opus 4.870.10% ± 1.16

Claude Sonnet 566.41% ± 1.33

Key takeaways

Citation validity separates the leading systems more clearly than answer fluency.
Explicit abstention improves safety without reducing overall task completion.
Performance drops on mixed-jurisdiction prompts and long procedural histories.

Methodology

Every system is run against the same frozen task set and scorer. Results are the mean across repeated trials; uncertainty is reported as the standard error of the mean. Costs use public API pricing at evaluation time and latency is measured end to end.

Evaluation items: 420
Repeated runs: 3–6 / system

Updates