How we measure

Task Design

Our benchmarks reflect the complexity of real-world tasks, which necessitates evaluating multiple types of capabilities:

• Tool-Use: How well can models call the right tools to solve problems?
• Multiple Modalities: How well can models handle images, tabular data, files, and other modalities beyond text?
• Reasoning: Models are increasingly trained to output reasoning before answering; do these capabilities actually improve real-world utility?
• Long-Context Capabilities: Can models reason over long contexts, such as extensive legal documents or large codebases?
• Long-Horizon Tasks: Can models autonomously work on tasks that take minutes, hours, or longer?

Public and Private Sets

A major problem with evaluations of AI models is test-set leakage. Benchmark data can contaminate training sets either directly or through synthetic data, undermining the validity of reported results. Thus, we offer private benchmarking; for transparency and fairness, we offer (for most benchmarks):

• Public Validation Set: A completely open dataset, to provide transparency in the types of samples we use for evaluation.
• Private Validation Set: A larger, privately held dataset, which we license for companies to do their own internal validation. We provide statistical proof that this is correlated with our test suite.
• Test Set: This dataset remains private at all times, and is the only dataset that is used for the benchmarks we publish.

Metrics and Evaluation

• Accuracy: Strict accuracy checks, as well as rubric-based LLM-as-a-judge metrics.
• Latency: Time for a model to return a complete response.
• Cost: Operational cost of running each model from an API provider.
• Additional insights: Tool-use statistics, qualitative error analysis, and comparisons between models.

Error Bars

We report standard errors alongside benchmark scores to reflect statistical uncertainty. For single-run benchmarks, error bars are the standard error of the mean over instance-level scores. For multiple-run benchmarks, we estimate uncertainty over runs. For composite benchmarks, we propagate uncertainty from each component using weighted variance pooling.

Evaluating agentic systems

Since models are often used as part of agentic systems with general scaffolds, and part of larger workflows or products, it is important to design evaluations that measure capabilities of these kinds of systems. Our benchmarks test tool-calling, multi-turn flows, coding skills, and computer-use.