Overview
Our benchmarks measure capability and reliability on realistic tasks. Every public number points to a versioned benchmark, immutable model configuration, scoring method, source record, and reviewed release.
Task Design
Real work mixes several capabilities:
- Tool use: selecting and operating the right tools.
- Multiple modalities: reasoning across text, images, files, and tables.
- Long context: maintaining evidence across large work products.
- Long horizon: completing tasks that take minutes or hours, not seconds.
Public and Private Sets
Public items support transparency; private held-out items limit test-set leakage. Dataset manifests record item hashes and licensing without exposing gold answers.
Metrics and Evaluation
Metrics are defined before evaluation. Composite indexes store component weights, normalization rules, missing-data policies, and scoring engine versions.
Error Bars
Point estimates are published with standard error or confidence intervals, sample size, trial count, and coverage. Rankings use an explicit tie policy.
Evaluating Systems and Scaffolds
We preserve the complete model system: serving provider, parameters, tools, runner and scorer commits, container digests, prompts, and price cards.
Work with us
Tell us what you need to measure. We’ll help shape a rigorous evaluation plan.