PLPramaana

LiveCodeBench

Updated 7/4/2026
Academic

Contamination-resistant coding problems collected continuously from recent programming competitions.

Model leaderboard

1GPT-5.6 Sol
Run settings: temperature 0, max tokens 32768, extended reasoning, 6 repeated runs.		83.00% ± 0.99$3.50793s
2Claude Fable 5
Run settings: temperature 0, max tokens 32768, extended reasoning, 5 repeated runs.		79.21% ± 0.82$2.90734s
3Claude Opus 4.8
Run settings: temperature 0, max tokens 32768, extended reasoning, 3 repeated runs.		76.92% ± 1.16$2.101410s
4Claude Sonnet 5
Run settings: temperature 0, max tokens 32768, extended reasoning, 4 repeated runs.		76.53% ± 1.33$1.85662s
5DeepSeek V4 ProOPEN
Run settings: temperature 0.2, max tokens 32768, standard reasoning, 3 repeated runs.		73.07% ± 1.16$0.24537s
6GLM 5.2OPEN
Run settings: temperature 0.2, max tokens 32768, standard reasoning, 4 repeated runs.		73.01% ± 0.65$0.37543s
7Kimi K2.7 CodeOPEN
Run settings: temperature 0.2, max tokens 32768, standard reasoning, 4 repeated runs.		72.96% ± 1.33$0.29490s
8Qwen 3.7 PlusOPEN
Run settings: temperature 0.2, max tokens 32768, standard reasoning, 6 repeated runs.		72.51% ± 0.99$0.32444s
9GPT-5.5
Run settings: temperature 0, max tokens 32768, standard reasoning, 6 repeated runs.		72.29% ± 1.67$2.63723s
10Muse Spark 1.1OPEN
Run settings: temperature 0.2, max tokens 32768, extended reasoning, 5 repeated runs.		72.29% ± 1.50$0.17412s

Results reflect a fixed evaluation snapshot. API prices and provider behavior may change after the recorded run date.

License: Original dataset terms

Motivation

Generic leaderboards rarely show whether a model can complete the work people actually depend on. This evaluation turns a real workflow into a controlled, repeatable test with visible uncertainty, cost, and latency.

Results

Industry average accuracy comparison

Mean score with standard-error whiskers across repeated runs.

GPT-5.6 Sol83.00% ± 0.99
Claude Fable 579.21% ± 0.82
Claude Opus 4.876.92% ± 1.16

Key takeaways

  • Recent hard problems expose a larger frontier gap than static coding sets.
  • Execution feedback helps most on medium-difficulty problems.
  • Pass@1 and cost efficiency favor different systems.

Methodology

Every system is run against the same frozen task set and scorer. Results are the mean across repeated trials; uncertainty is reported as the standard error of the mean. Costs use public API pricing at evaluation time and latency is measured end to end.

Evaluation items
1,040
Repeated runs
3–6 / system
Visit the original benchmark source

Updates

  1. 7/4/2026

    Latest frontier model results added.

  2. 5/12/2026

    Scorer calibration and refusal handling updated.

  3. 2/18/2026

    Evaluation set published.