PLPramaana

Pramaana Index

Updated 7/9/2026
Pramaana Original

A weighted measure of frontier-model performance across the professional and technical work represented in Pramaana's versioned, reproducible evaluation suite.

Model leaderboard

1Claude Fable 5
Run settings: temperature 0, max tokens 16384, extended reasoning, 3 repeated runs.		73.72% ± 0.48$2.90471s
2GPT-5.6 Sol
Run settings: temperature 0, max tokens 16384, extended reasoning, 4 repeated runs.		71.71% ± 0.65$3.50509s
3Claude Opus 4.8
Run settings: temperature 0, max tokens 16384, extended reasoning, 5 repeated runs.		69.90% ± 0.82$2.10905s
4Claude Sonnet 5
Run settings: temperature 0, max tokens 16384, extended reasoning, 6 repeated runs.		68.61% ± 0.99$1.85425s
5Muse Spark 1.1OPEN
Run settings: temperature 0.2, max tokens 16384, extended reasoning, 3 repeated runs.		66.80% ± 1.16$0.17264s
6GPT-5.5
Run settings: temperature 0, max tokens 16384, standard reasoning, 4 repeated runs.		66.80% ± 1.33$2.63464s
7GLM 5.2OPEN
Run settings: temperature 0.2, max tokens 16384, standard reasoning, 6 repeated runs.		64.79% ± 1.67$0.37348s
8Grok 4.5
Run settings: temperature 0, max tokens 16384, standard reasoning, 5 repeated runs.		64.61% ± 1.50$1.08264s
9Gemini 3.5 Flash
Run settings: temperature 0, max tokens 16384, standard reasoning, 3 repeated runs.		64.46% ± 0.48$0.40164s
10Qwen 3.7 PlusOPEN
Run settings: temperature 0.2, max tokens 16384, standard reasoning, 4 repeated runs.		62.52% ± 0.65$0.32285s

Results reflect a fixed evaluation snapshot. API prices and provider behavior may change after the recorded run date.

License: Research preview

Motivation

AI capabilities are advancing faster than most organizations can update their deployment decisions. Leaders need evidence about where systems help, where they fail, and what they cost before a deployment can be responsible. The Pramaana Index measures practical performance across real workflows drawn from high-impact economic sectors, using versioned task sets, fixed system configurations, and repeatable scoring. It also keeps evidence about failure modes visible to reviewers.

A single score never tells the whole story. The index preserves operational trade-offs by publishing task success beside uncertainty, test cost, latency, refusal behavior, and the exact system settings used for every run. This makes each ranking auditable and lets teams choose a model for the work they actually need to perform. Every result remains tied to its benchmark version and raw evidence, so reviewers can reconstruct decisions instead of trusting a single summary number.

Results

Industry average accuracy comparison

Mean score with standard-error whiskers across repeated runs.

Claude Fable 573.72% ± 0.48
GPT-5.6 Sol71.71% ± 0.65
Claude Opus 4.869.90% ± 0.82

Key takeaways

  • The strongest aggregate model is not the leader on every domain; model selection remains workload-specific.
  • Open-weight systems now sit on the efficient frontier for several coding and education tasks.
  • Repeated trials materially change rank order on high-variance agentic evaluations.

Methodology

Benchmarks are selected for economic relevance, scoring reliability, and resistance to contamination. Domain weights are reviewed each quarter against Pramaana's task taxonomy.

Scores are normalized within each benchmark before aggregation. Missing evaluations receive no imputed advantage and are excluded from the applicable sub-index denominator.

Evaluation items
1,280
Repeated runs
3–6 / system
Pramaana Index = Σ (domain weight × normalized task score)
95% CI = score ± 1.96 × standard error

Updates

  1. 7/9/2026

    Latest frontier model results added.

  2. 5/12/2026

    Scorer calibration and refusal handling updated.

  3. 2/18/2026

    Evaluation set published.