Pramaana Multimodal Index

Updated 7/8/2026

Pramaana Original

Measures how reliably models reason across documents, images, spreadsheets, audio, and mixed-modality professional tasks.

View:Task:


1Claude Fable 5 Run settings: temperature 0, max tokens 16384, extended reasoning, 4 repeated runs.	71.71% ± 0.65	$3.64	602s
2GPT-5.6 Sol Run settings: temperature 0, max tokens 16384, extended reasoning, 5 repeated runs.	69.70% ± 0.82	$4.40	651s
3Claude Opus 4.8 Run settings: temperature 0, max tokens 16384, extended reasoning, 6 repeated runs.	65.82% ± 0.99	$2.64	1158s
4GPT-5.5 Run settings: temperature 0, max tokens 16384, standard reasoning, 5 repeated runs.	64.79% ± 1.50	$3.31	593s
5Muse Spark 1.1OPEN Run settings: temperature 0.2, max tokens 16384, extended reasoning, 4 repeated runs.	64.79% ± 1.33	$0.22	338s
6Claude Sonnet 5 Run settings: temperature 0, max tokens 16384, extended reasoning, 3 repeated runs.	64.53% ± 1.16	$2.32	543s
7Grok 4.5 Run settings: temperature 0, max tokens 16384, standard reasoning, 6 repeated runs.	62.60% ± 1.67	$1.36	338s
8Gemini 3.5 Flash Run settings: temperature 0, max tokens 16384, standard reasoning, 4 repeated runs.	62.45% ± 0.65	$0.51	210s
9GLM 5.2OPEN Run settings: temperature 0.2, max tokens 16384, standard reasoning, 3 repeated runs.	60.71% ± 0.48	$0.47	446s
10Qwen 3.7 PlusOPEN Run settings: temperature 0.2, max tokens 16384, standard reasoning, 5 repeated runs.	60.51% ± 0.82	$0.40	364s

Motivation

Generic leaderboards rarely show whether a model can complete the work people actually depend on. This evaluation turns a real workflow into a controlled, repeatable test with visible uncertainty, cost, and latency.

Results

Industry average accuracy comparison

Mean score with standard-error whiskers across repeated runs.

Claude Fable 571.71% ± 0.65

GPT-5.6 Sol69.70% ± 0.82

Claude Opus 4.865.82% ± 0.99

Key takeaways

Document understanding remains more reliable than spreadsheet manipulation.
Long context alone does not predict cross-file reasoning quality.
Multimodal latency varies by more than 5× among similarly accurate systems.

Methodology

Every system is run against the same frozen task set and scorer. Results are the mean across repeated trials; uncertainty is reported as the standard error of the mean. Costs use public API pricing at evaluation time and latency is measured end to end.

Evaluation items: 860
Repeated runs: 3–6 / system

Updates