PLPramaana

GPQA Diamond

Updated 7/9/2026
Academic

Graduate-level, Google-proof questions that require deep reasoning across biology, chemistry, and physics.

Model leaderboard

1GPT-5.6 Sol
Run settings: temperature 0, max tokens 16384, extended reasoning, 6 repeated runs.		88.92% ± 1.67$4.40793s
2Claude Fable 5
Run settings: temperature 0, max tokens 16384, extended reasoning, 5 repeated runs.		87.60% ± 1.50$3.64734s
3GPT-5.5
Run settings: temperature 0, max tokens 16384, standard reasoning, 6 repeated runs.		81.91% ± 0.99$3.31723s
4Claude Opus 4.8
Run settings: temperature 0, max tokens 16384, extended reasoning, 3 repeated runs.		81.71% ± 0.48$2.641410s
5Muse Spark 1.1OPEN
Run settings: temperature 0.2, max tokens 16384, extended reasoning, 5 repeated runs.		80.68% ± 0.82$0.22412s
6Claude Sonnet 5
Run settings: temperature 0, max tokens 16384, extended reasoning, 4 repeated runs.		80.42% ± 0.65$2.32662s
7Grok 4.5
Run settings: temperature 0, max tokens 16384, standard reasoning, 3 repeated runs.		79.62% ± 1.16$1.36412s
8Gemini 3.5 Flash
Run settings: temperature 0, max tokens 16384, standard reasoning, 5 repeated runs.		76.84% ± 1.50$0.51256s
9GLM 5.2OPEN
Run settings: temperature 0.2, max tokens 16384, standard reasoning, 4 repeated runs.		76.60% ± 1.33$0.47543s
10Qwen 3.7 PlusOPEN
Run settings: temperature 0.2, max tokens 16384, standard reasoning, 6 repeated runs.		76.40% ± 1.67$0.40444s

Results reflect a fixed evaluation snapshot. API prices and provider behavior may change after the recorded run date.

License: Original dataset terms

Motivation

Generic leaderboards rarely show whether a model can complete the work people actually depend on. This evaluation turns a real workflow into a controlled, repeatable test with visible uncertainty, cost, and latency.

Results

Industry average accuracy comparison

Mean score with standard-error whiskers across repeated runs.

GPT-5.6 Sol88.92% ± 1.67
Claude Fable 587.60% ± 1.50
GPT-5.581.91% ± 0.99

Key takeaways

  • Reasoning-time scaling continues to improve performance on expert questions.
  • Physics has the highest between-run variance in the evaluated set.
  • Rank differences under one percentage point are generally not significant.

Methodology

Every system is run against the same frozen task set and scorer. Results are the mean across repeated trials; uncertainty is reported as the standard error of the mean. Costs use public API pricing at evaluation time and latency is measured end to end.

Evaluation items
198
Repeated runs
3–6 / system
Visit the original benchmark source

Updates

  1. 7/9/2026

    Latest frontier model results added.

  2. 5/12/2026

    Scorer calibration and refusal handling updated.

  3. 2/18/2026

    Evaluation set published.