PLPramaana

MMLU-Pro

Updated 6/18/2026
Academic

A more challenging and reasoning-focused extension of MMLU spanning fourteen knowledge domains.

Model leaderboard

1GPT-5.6 Sol
Run settings: temperature 0, max tokens 16384, extended reasoning, 3 repeated runs.		86.41% ± 0.48$5.30936s
2Claude Fable 5
Run settings: temperature 0, max tokens 16384, extended reasoning, 6 repeated runs.		83.02% ± 1.67$4.39866s
3GPT-5.5
Run settings: temperature 0, max tokens 16384, standard reasoning, 3 repeated runs.		79.40% ± 1.16$3.99852s
4Claude Opus 4.8
Run settings: temperature 0, max tokens 16384, extended reasoning, 4 repeated runs.		79.20% ± 0.65$3.181663s
5Claude Sonnet 5
Run settings: temperature 0, max tokens 16384, extended reasoning, 5 repeated runs.		77.91% ± 0.82$2.80780s
6Grok 4.5
Run settings: temperature 0, max tokens 16384, standard reasoning, 4 repeated runs.		77.11% ± 1.33$1.64486s
7Muse Spark 1.1OPEN
Run settings: temperature 0.2, max tokens 16384, extended reasoning, 6 repeated runs.		76.10% ± 0.99$0.26486s
8GLM 5.2OPEN
Run settings: temperature 0.2, max tokens 16384, standard reasoning, 5 repeated runs.		74.09% ± 1.50$0.56640s
9Gemini 3.5 Flash
Run settings: temperature 0, max tokens 16384, standard reasoning, 6 repeated runs.		72.26% ± 1.67$0.61301s
10Qwen 3.7 PlusOPEN
Run settings: temperature 0.2, max tokens 16384, standard reasoning, 3 repeated runs.		71.82% ± 0.48$0.48524s

Results reflect a fixed evaluation snapshot. API prices and provider behavior may change after the recorded run date.

License: Original dataset terms

Motivation

Generic leaderboards rarely show whether a model can complete the work people actually depend on. This evaluation turns a real workflow into a controlled, repeatable test with visible uncertainty, cost, and latency.

Results

Industry average accuracy comparison

Mean score with standard-error whiskers across repeated runs.

GPT-5.6 Sol86.41% ± 0.48
Claude Fable 583.02% ± 1.67
GPT-5.579.40% ± 1.16

Key takeaways

  • Performance is most stable in high-sample subject categories.
  • Business and law remain stronger than physics for general-purpose systems.
  • The open-weight gap has narrowed substantially year over year.

Methodology

Every system is run against the same frozen task set and scorer. Results are the mean across repeated trials; uncertainty is reported as the standard error of the mean. Costs use public API pricing at evaluation time and latency is measured end to end.

Evaluation items
12,032
Repeated runs
3–6 / system
Visit the original benchmark source

Updates

  1. 6/18/2026

    Latest frontier model results added.

  2. 5/12/2026

    Scorer calibration and refusal handling updated.

  3. 2/18/2026

    Evaluation set published.