Mercury 2.5: benchmarks and ranking

How the ranking works

1. Collect — Official data from the 11 leaderboards below is fetched every hour. Scores marked as estimates are not used.
2. Match — Different spellings of the same model are merged. For variants such as reasoning effort, each site’s best result is used.
3. Standardize — Each site’s main metric is standardized within that site, keeping score gaps rather than just ranks.
4. Equate — A factor model, bridged by models listed on several sites, corrects for each site’s difficulty and model mix. Noisier sites get less weight.
5. No penalty for missing data — Models are scored only from the sites that rated them. Uncertainty is shown as a 90% interval. Models rated on 2+ sites are ranked.
6. Arena — Arena (formerly LMArena) is used for domain rankings only, not for the overall score. New models are added automatically.