Events

Date: 22 May 2025   Time: 16:00 - 17:00

Location: Ada Lovelace meeting room (Alan Turing Institute) & online

This Thursday on 22nd May at 4-5pm (GMT), we will have an event hosted online and at the Alan Turing Institute with Shivalika Singh (Research Scientist) and Yiyang Nan (Research Scholar) from the Cohere Open Science Research Team, who will be presenting their talk "The Leaderboard Illusion".

Date: Thursday, 22nd May
Time: 4-5pm (GMT)
Location: Ada Lovelace meeting room (Alan Turing Institute) & online
Zoom: turing-uk.zoom.us/j/6687887536

Title: The Leaderboard Illusion
Paper: arxiv.org/abs/2504.20879
Presenters: Shivalika Singh & Yiyang Nan

Abstract:
The Chatbot Arena, a widely used benchmark for ranking large language models, has significant systemic issues that distort fair evaluation. The study reveals that private testing and selective disclosure practices, such as Meta testing 27 LLM variants before releasing LLaMA-4, give major providers like OpenAI and Google an unfair advantage. These companies receive disproportionately high exposure on the platform, with around 20% of total data each, while 83 open-weight models collectively receive less than 30%. This data access imbalance enables performance gains of up to 112% through overfitting to Arena-specific dynamics, rather than reflecting true model quality. The authors call for reforms to ensure transparency and fairness in benchmarking practices.