PLPramaana

MATH-500

Updated 6/20/2026
Academic

A curated subset of challenging competition mathematics problems with exact-answer grading.

Model leaderboard

1GPT-5.6 Sol
Run settings: temperature 0, max tokens 16384, extended reasoning, 5 repeated runs.		84.71% ± 1.50$3.50651s
2Claude Fable 5
Run settings: temperature 0, max tokens 16384, extended reasoning, 4 repeated runs.		81.52% ± 1.33$2.90602s
3GPT-5.5
Run settings: temperature 0, max tokens 16384, standard reasoning, 5 repeated runs.		78.80% ± 0.82$2.63593s
4Grok 4.5
Run settings: temperature 0, max tokens 16384, standard reasoning, 6 repeated runs.		77.91% ± 0.99$1.08338s
5Claude Opus 4.8
Run settings: temperature 0, max tokens 16384, extended reasoning, 6 repeated runs.		77.70% ± 1.67$2.101158s
6Claude Sonnet 5
Run settings: temperature 0, max tokens 16384, extended reasoning, 3 repeated runs.		76.41% ± 0.48$1.85543s
7DeepSeek V4 ProOPEN
Run settings: temperature 0.2, max tokens 16384, standard reasoning, 6 repeated runs.		74.98% ± 1.67$0.24441s
8Muse Spark 1.1OPEN
Run settings: temperature 0.2, max tokens 16384, extended reasoning, 4 repeated runs.		74.60% ± 0.65$0.17338s
9Qwen 3.7 PlusOPEN
Run settings: temperature 0.2, max tokens 16384, standard reasoning, 5 repeated runs.		73.42% ± 1.50$0.32364s
10GLM 5.2OPEN
Run settings: temperature 0.2, max tokens 16384, standard reasoning, 3 repeated runs.		72.59% ± 1.16$0.37446s

Results reflect a fixed evaluation snapshot. API prices and provider behavior may change after the recorded run date.

License: Original dataset terms

Motivation

Generic leaderboards rarely show whether a model can complete the work people actually depend on. This evaluation turns a real workflow into a controlled, repeatable test with visible uncertainty, cost, and latency.

Results

Industry average accuracy comparison

Mean score with standard-error whiskers across repeated runs.

GPT-5.6 Sol84.71% ± 1.50
Claude Fable 581.52% ± 1.33
GPT-5.578.80% ± 0.82

Key takeaways

  • Extended reasoning provides the largest lift on geometry and counting.
  • Independent verification sharply lowers confident arithmetic errors.
  • Open-weight leaders are within the uncertainty band of several closed systems.

Methodology

Every system is run against the same frozen task set and scorer. Results are the mean across repeated trials; uncertainty is reported as the standard error of the mean. Costs use public API pricing at evaluation time and latency is measured end to end.

Evaluation items
500
Repeated runs
3–6 / system
Visit the original benchmark source

Updates

  1. 6/20/2026

    Latest frontier model results added.

  2. 5/12/2026

    Scorer calibration and refusal handling updated.

  3. 2/18/2026

    Evaluation set published.