PLPramaana

SWE-bench Verified

Updated 7/6/2026
Academic

Measures whether autonomous coding systems can resolve human-validated issues from real GitHub repositories.

Model leaderboard

1GPT-5.6 Sol
Run settings: temperature 0, max tokens 32768, extended reasoning, 5 repeated runs.		64.02% ± 0.82$7.10651s
2Claude Fable 5
Run settings: temperature 0, max tokens 32768, extended reasoning, 4 repeated runs.		62.30% ± 0.65$5.88602s
3Claude Opus 4.8
Run settings: temperature 0, max tokens 32768, extended reasoning, 6 repeated runs.		60.01% ± 0.99$4.261158s
4Claude Sonnet 5
Run settings: temperature 0, max tokens 32768, extended reasoning, 3 repeated runs.		59.62% ± 1.16$3.75543s
5GLM 5.2OPEN
Run settings: temperature 0.2, max tokens 32768, standard reasoning, 3 repeated runs.		56.10% ± 0.48$0.76446s
6Kimi K2.7 CodeOPEN
Run settings: temperature 0.2, max tokens 32768, standard reasoning, 3 repeated runs.		56.05% ± 1.16$0.59402s
7Qwen 3.7 PlusOPEN
Run settings: temperature 0.2, max tokens 32768, standard reasoning, 5 repeated runs.		55.60% ± 0.82$0.65364s
8Muse Spark 1.1OPEN
Run settings: temperature 0.2, max tokens 32768, extended reasoning, 4 repeated runs.		55.38% ± 1.33$0.35338s
9DeepSeek V4 ProOPEN
Run settings: temperature 0.2, max tokens 32768, standard reasoning, 6 repeated runs.		54.09% ± 0.99$0.49441s
10GPT-5.5
Run settings: temperature 0, max tokens 32768, standard reasoning, 5 repeated runs.		53.31% ± 1.50$5.34593s

Results reflect a fixed evaluation snapshot. API prices and provider behavior may change after the recorded run date.

License: Original dataset terms

Motivation

Generic leaderboards rarely show whether a model can complete the work people actually depend on. This evaluation turns a real workflow into a controlled, repeatable test with visible uncertainty, cost, and latency.

Results

Industry average accuracy comparison

Mean score with standard-error whiskers across repeated runs.

GPT-5.6 Sol64.02% ± 0.82
Claude Fable 562.30% ± 0.65
Claude Opus 4.860.01% ± 0.99

Key takeaways

  • Repository navigation and test interpretation drive most remaining errors.
  • More tokens do not reliably improve resolved rate after the first agent retry.
  • Code-specialized open models offer the lowest cost per resolved issue.

Methodology

Every system is run against the same frozen task set and scorer. Results are the mean across repeated trials; uncertainty is reported as the standard error of the mean. Costs use public API pricing at evaluation time and latency is measured end to end.

Evaluation items
500
Repeated runs
3–6 / system
Visit the original benchmark source

Updates

  1. 7/6/2026

    Latest frontier model results added.

  2. 5/12/2026

    Scorer calibration and refusal handling updated.

  3. 2/18/2026

    Evaluation set published.