Open ASR Leaderboard adds Egyptian and Nigerian English accent benchmarks

Hugging Face has expanded its Open ASR Leaderboard with two new private benchmark datasets contributed through a collaboration with Liva AI, covering Egyptian and Nigerian English accents. The additions bring 1.34 hours of conversational Egyptian English and 1.55 hours of conversational Nigerian English, both with punctuated, cased transcriptions that include disfluencies. These datasets join existing private splits from Appen, DataOceanAI, and Voice Arena that were added earlier this year to provide held-out evaluation data and reduce benchmark gaming.

The leaderboard now organizes accent breakdowns using Kachru's Three Circles framework, comparing US versus non-US accents and distinguishing Inner Circle varieties (US, UK, Australia, Canada) from Outer and Expanding Circle varieties (India, Nigeria, Egypt). This structural change aims to improve representation in automatic speech recognition evaluation and provide more nuanced performance reporting as more accents are added.

Dataset structure and evaluation methodology

• Dataset composition: Liva AI Conversational EG provides 1.34 hours of Egyptian English, while Liva AI Conversational NG provides 1.55 hours of Nigerian English, both in spontaneous conversational style.
• Transcription fidelity: Both datasets include punctuated, cased transcriptions with disfluencies preserved, enabling evaluation of ASR systems on more naturalistic speech patterns than scripted read speech.
• Privacy and anti-gaming: The datasets are held private and not downloadable, following the leaderboard's strategy of minimizing benchmaxxing by preventing model training on evaluation data.
• Accent taxonomy: The leaderboard now reports results using Kachru's Three Circles model, separating Inner Circle English varieties from Outer and Expanding Circle varieties, with plans to further subdivide the latter category as more accents are added.
• Data collection model: Liva AI gathers multimodal interaction data through consumer products in real-world use, capturing natural high-context behavior rather than artificially elicited speech.

Why it matters

The expansion of the Open ASR Leaderboard to include Egyptian and Nigerian English addresses a persistent gap in speech recognition evaluation, where models are typically benchmarked primarily on US and UK accents and perform poorly on Global South varieties. For developers and researchers working on open-source ASR systems, these held-out private benchmarks provide trustworthy metrics to identify performance disparities and guide model improvements for underrepresented speaker populations.