Disclaimer
Rankings are for reference only. Actual model performance depends on the use case, task, prompting, and many other factors. These rankings reflect relative performance only on the benchmarks in our dataset and do not represent overall capability in every setting. Consider multiple evaluations in the context of your own requirements.
Frontier model rankings use Bradley–Terry / Elo pairwise scoring. Results within each benchmark are converted into pairwise comparisons and then fitted into a consolidated score. The method, data sources, and known limitations are described below.
Algorithm
We convert each benchmark ranking into pairwise outcomes, then fit relative strength on an Elo scale with a Bradley–Terry model:
- Within each benchmark, participating models are sorted by result and paired against one another, with the higher-scoring model treated as the winner.
- A benchmark normally has a weight of 1.0. A benchmark with one or no effective independent sources is weighted at 0.5. Arena.ai is exempt because it aggregates a large volume of real-user votes.
- Equal results are treated as ties, and 0.5 smoothing is added to the outcome matrix to avoid infinite scores for small samples or undefeated models.
- Expected pairwise win probability uses the Elo logistic function:
E = 1 / (1 + 10^((R_opponent - R_self) / 400)) - All pairwise results are fitted in batch until convergence or a limit of 500 iterations is reached.
- Each connected component is centered at 1500, preventing model groups with no direct comparison from acquiring a misleading absolute gap.
Key properties
- Defeating a stronger opponent contributes more than defeating a weaker one.
- Models can receive a meaningful score with limited benchmark coverage; coverage volume does not dominate the score.
- Benchmarks with fewer sources still contribute, but with reduced influence on the final result.
- Only result order is used; the size of the score margin is ignored.
- The calculation does not depend on a random seed or pair-processing order.
Data sources
Scores come from provider technical reports and public benchmark results, including:
- official technical reports from organizations such as Anthropic, OpenAI, Google, Zhipu AI, Alibaba, and DeepSeek;
- evaluations published by open model platforms such as ModelScope;
- the best result for each model on each benchmark across all collected sources;
- an “effective source,” defined as an independent source containing at least two indexed models eligible for ranking. Repeated snapshots from the same source count only once.
Known limitations
Selection bias
Providers tend to report benchmarks on which their models perform well. A model evaluated only on its strengths may receive an inflated score.
Score margins are ignored
Elo uses ordering, not the size of a result gap. A 0.1% win contributes the same outcome as a 50% win.
Protocol differences under the same benchmark name
Reports using the same benchmark name may differ in evaluation rules, prompt templates, judge models, sampled subsets, or execution settings. We retain source context wherever possible, but merged results cannot be guaranteed to come from identical conditions, so cross-report comparisons may be biased.
Coarse reliability weighting
Current reliability weighting has only two levels, 0.5 and 1.0, based on the number of effective sources. It cannot fully represent sample size, methodological quality, judge choice, coverage, or task relevance. Arena.ai’s exemption reflects only its aggregation of many real-user votes.
Scores are not comparable across domains
Benchmarks vary in difficulty and participant strength by domain, so absolute scores should not be compared directly across domains.