PLPramaana

Financial Modeling Agent

Updated 6/27/2026
Pramaana Original

Tests whether agents can audit, repair, and extend investment-banking and corporate-finance spreadsheet models.

Model leaderboard

1Claude Fable 5
Run settings: temperature 0, max tokens 16384, extended reasoning, 6 repeated runs.		68.40% ± 0.99$5.13866s
2GPT-5.6 Sol
Run settings: temperature 0, max tokens 16384, extended reasoning, 3 repeated runs.		61.82% ± 1.16$6.20936s
3Claude Opus 4.8
Run settings: temperature 0, max tokens 16384, extended reasoning, 4 repeated runs.		60.01% ± 1.33$3.721663s
4Muse Spark 1.1OPEN
Run settings: temperature 0.2, max tokens 16384, extended reasoning, 6 repeated runs.		58.98% ± 1.67$0.31486s
5Claude Sonnet 5
Run settings: temperature 0, max tokens 16384, extended reasoning, 5 repeated runs.		58.72% ± 1.50$3.27780s
6GLM 5.2OPEN
Run settings: temperature 0.2, max tokens 16384, standard reasoning, 5 repeated runs.		57.30% ± 0.82$0.66640s
7GPT-5.5
Run settings: temperature 0, max tokens 16384, standard reasoning, 3 repeated runs.		56.91% ± 0.48$4.66852s
8Gemini 3.5 Flash
Run settings: temperature 0, max tokens 16384, standard reasoning, 6 repeated runs.		55.14% ± 0.99$0.71301s
9Grok 4.5
Run settings: temperature 0, max tokens 16384, standard reasoning, 4 repeated runs.		54.72% ± 0.65$1.92486s
10Qwen 3.7 PlusOPEN
Run settings: temperature 0.2, max tokens 16384, standard reasoning, 3 repeated runs.		54.70% ± 1.16$0.57524s

Results reflect a fixed evaluation snapshot. API prices and provider behavior may change after the recorded run date.

License: Research preview

Motivation

Generic leaderboards rarely show whether a model can complete the work people actually depend on. This evaluation turns a real workflow into a controlled, repeatable test with visible uncertainty, cost, and latency.

Results

Industry average accuracy comparison

Mean score with standard-error whiskers across repeated runs.

Claude Fable 568.40% ± 0.99
GPT-5.6 Sol61.82% ± 1.16
Claude Opus 4.860.01% ± 1.33

Key takeaways

  • Formula correctness and presentation quality are weakly correlated.
  • Tool retries improve success but can triple per-test cost.
  • The best models still miss cross-sheet assumptions in dense workbooks.

Methodology

Every system is run against the same frozen task set and scorer. Results are the mean across repeated trials; uncertainty is reported as the standard error of the mean. Costs use public API pricing at evaluation time and latency is measured end to end.

Evaluation items
310
Repeated runs
3–6 / system

Updates

  1. 6/27/2026

    Latest frontier model results added.

  2. 5/12/2026

    Scorer calibration and refusal handling updated.

  3. 2/18/2026

    Evaluation set published.