PLPramaana

Agentic Reliability

Updated 7/8/2026
Pramaana Original

Stress-tests recovery, state tracking, and safe escalation across long-running tool-using tasks.

Model leaderboard

1Claude Fable 5
Run settings: temperature 0, max tokens 16384, extended reasoning, 6 repeated runs.		59.91% ± 0.99$3.64866s
2GPT-5.6 Sol
Run settings: temperature 0, max tokens 16384, extended reasoning, 3 repeated runs.		57.90% ± 1.16$4.40936s
3Claude Opus 4.8
Run settings: temperature 0, max tokens 16384, extended reasoning, 4 repeated runs.		54.02% ± 1.33$2.641663s
4GPT-5.5
Run settings: temperature 0, max tokens 16384, standard reasoning, 3 repeated runs.		52.99% ± 0.48$3.31852s
5Muse Spark 1.1OPEN
Run settings: temperature 0.2, max tokens 16384, extended reasoning, 6 repeated runs.		52.99% ± 1.67$0.22486s
6Claude Sonnet 5
Run settings: temperature 0, max tokens 16384, extended reasoning, 5 repeated runs.		52.73% ± 1.50$2.32780s
7Grok 4.5
Run settings: temperature 0, max tokens 16384, standard reasoning, 4 repeated runs.		50.80% ± 0.65$1.36486s
8Gemini 3.5 Flash
Run settings: temperature 0, max tokens 16384, standard reasoning, 6 repeated runs.		49.15% ± 0.99$0.51301s
9GLM 5.2OPEN
Run settings: temperature 0.2, max tokens 16384, standard reasoning, 5 repeated runs.		48.91% ± 0.82$0.47640s
10Qwen 3.7 PlusOPEN
Run settings: temperature 0.2, max tokens 16384, standard reasoning, 3 repeated runs.		48.71% ± 1.16$0.40524s

Results reflect a fixed evaluation snapshot. API prices and provider behavior may change after the recorded run date.

License: Research preview

Motivation

Generic leaderboards rarely show whether a model can complete the work people actually depend on. This evaluation turns a real workflow into a controlled, repeatable test with visible uncertainty, cost, and latency.

Results

Industry average accuracy comparison

Mean score with standard-error whiskers across repeated runs.

Claude Fable 559.91% ± 0.99
GPT-5.6 Sol57.90% ± 1.16
Claude Opus 4.854.02% ± 1.33

Key takeaways

  • Recovery behavior is a stronger deployment predictor than first-attempt success.
  • Silent state loss remains common after tool errors.
  • Explicit escalation policies improve safety and completion rate.

Methodology

Every system is run against the same frozen task set and scorer. Results are the mean across repeated trials; uncertainty is reported as the standard error of the mean. Costs use public API pricing at evaluation time and latency is measured end to end.

Evaluation items
240
Repeated runs
3–6 / system

Updates

  1. 7/8/2026

    Latest frontier model results added.

  2. 5/12/2026

    Scorer calibration and refusal handling updated.

  3. 2/18/2026

    Evaluation set published.