Open ASR Leaderboard: Hindi & Indian English Monsoon Update

A leaderboard can tell you that two speech recognizers are nearly tied. Monsoon adds the missing question: tied for which speakers, regions, devices, and writing conventions? The new en-IN and hi-IN evaluation sets make the Open ASR Leaderboard more useful for multilingual ASR evaluation, but their short public recordings still require careful interpretation.

The short answer: Monsoon changes what the leaderboard can explain

The Open ASR Leaderboard now includes two Voice Arena evaluation sets: Monsoon en-IN for Indian English and Monsoon hi-IN for Hindi. Indian English appears in the main leaderboard columns and contributes to the headline Average WER. Hindi appears in the multilingual evaluation view, where a model must cover the selected languages to receive a multilingual ranking.

The four public and private splits contain 4,888 speaker-disjoint speakers and record 12 speaker-related attributes across demographics, background, geography, and handsets. Researchers can therefore move beyond “Which model has the lowest average WER?” and inspect where each model is less stable. Monsoon is a meaningful evaluation upgrade for regional robustness and Hindi transcript scoring, not a production guarantee or a nationally representative measure of every Indian language.

What Monsoon adds to the Open ASR Leaderboard

Monsoon adds four test splits collected through the same conversational, spontaneous-speech pipeline. Each language has a public split for self-scoring and a private split intended to make benchmark-specific tuning harder. The official announcement describes the release as the first Indian-language addition to this leaderboard's multilingual context.

The split-by-split inventory

The public Monsoon en-IN dataset card reports 16 kHz audio, a 9.6-second mean clip length, and a 10.4-second median. The Monsoon hi-IN card reports a 6.4-second mean and a 5.0-second median for its public split. The four splits total 17.00 hours, so this is a diversity-focused benchmark built from short conversational segments rather than long-form calls.

Public for reproduction, private for generalization

Researchers can load the public splits from Hugging Face, run a recognizer, normalize its output, and compute a corpus-level score. For en-IN, the dataset card requires total insertion, deletion, and substitution counts to be divided by total reference words, rather than averaging per-clip WER.

For Hindi, the public card provides the lattice-aware voi-oiwer scorer. A minimal loading pattern is:

from datasets import load_dataset
from voi_oiwer import oiwer

hi = load_dataset(
    "VoiceArena/MonsoonASR-Open-ASR-leaderboard-hi-IN",
    split="test",
)
hypothesis = "replace with your model output for hi[0]"
result = oiwer(hypothesis, hi[0]["lattice"], "hindi")
score, (ins, dele, sub), ref_words = result[0], result[4], result[5]

The private splits remain held out and speaker-disjoint so a model team cannot tune against each evaluation example. The private-data design keeps those results in separate aggregate views instead of redefining the public default ranking.

Speaker coverage is the update's real evaluation change

The public en-IN set contains 1,444 speakers across 2,102 clips, while the public hi-IN set contains 468 speakers across 753 clips. The official announcement's coverage table reports that the ten largest contributors account for only 2.8% to 6.8% of total duration across the four splits.

This reduces the chance that one prolific speaker, microphone, or recording style dominates the score, but it does not make the sample representative. The 12 recorded attributes cover gender, date of birth, occupation, education, marital status, income, native district and state, current city, years in the current district, device manufacturer, and device model.

Indian English is a geographic distribution, not one accent

The public en-IN split spans 428 native districts, 30 states and union territories, 641 current cities, and 556 handset models. Its segment mix covers the South at 35%, East at 18%, Central at 18%, North at 16%, and West at 11%. No single state exceeds 13% of speakers, according to the Hugging Face announcement.

The fields native_district, native_state, current_city, years_spent_in_current_district, device_manufacturer, and device_model let evaluators compare regional and handset slices. “Indian English” is therefore not one microphone condition or one accent. A model can be strong on one regional speech distribution and less stable on another while retaining a competitive overall WER.

Hindi has different coverage—and a different scoring problem

The public Hindi split is smaller: 1.33 hours, 468 speakers, 202 districts, and 315 handset models. Hindi coverage is concentrated in the Hindi belt; the announcement says Uttar Pradesh accounts for about 40% of Hindi speakers in the collection. That is useful context for interpreting a Hindi result, but it does not show that every Hindi-speaking community is represented equally.

Hindi also contains English-origin words, code-switching, joined or separated compounds, Devanagari spelling variation, and Romanized alternatives. The hi-IN dataset card therefore stores a lattice in which each transcript span can contain multiple accepted written forms.

Regional error analysis exposes ties that aggregate WER hides

The official regional example uses the public Indian English split. Eight models fall between 4.81 and 4.99 WER, a spread of only 0.18 points. The announcement cautions that a roughly five-hour test set may not resolve a difference that small.

Once the same results are broken down by five geographic zones, the stability picture changes:

The wider spread reflects greater regional variability for Voxtral, not four times as many transcription errors. Regional weaknesses also differ by model, suggesting model-specific sensitivity without proving its cause.

A real user, @FoucqueThomas, highlighted the same distinction after the release:

“⚡ L'Open ASR Leaderboard ajoute l'hindi et l'anglais indien (4 888 locuteurs, 12 attributs/clip). Résultat : deux modèles quasi identiques sur le WER global affichent jusqu'à ~4x d'écart selon la région du locuteur. Un benchmark enfin honnête sur la diversité vocale.” — @FoucqueThomas on X

If users are concentrated in a particular region, an aggregate tie is not enough evidence to choose between models. The example does not prove that a particular training-data imbalance caused the regional pattern.

Why Hindi lattice scoring matters for multilingual ASR evaluation

Standard WER assumes a canonical reference, which works for casing and spelling normalization but breaks down for code-mixed Hindi. A valid output may use a Devanagari spelling, a Romanized form, an English-origin spelling, or a different compound-word boundary.

Monsoon hi-IN addresses this with reference lattices: the official release describes candidate variants being expanded, reviewed by native-speaking linguists, and filtered so that only forms consistent with the utterance remain. A valid alternative can count as correct instead of being penalized for differing from one annotator's rendering.

The leaderboard uses Orthographically-Informed Word Error Rate (OIWER) for the Hindi evaluation. An accepted variant counts as correct, while a genuine recognition error is still charged. The leaderboard integration commit shows the Hindi data and per-language multilingual views being wired into the application.

This scoring rule can change model ordering. The official announcement reports that flattening the Hindi lattices into single references raises error rates for every tested system and that the increases are not uniform. A multilingual leaderboard can therefore become more linguistically fair without reducing every result to one universal number.

How to use the updated leaderboard for model selection

Use Monsoon as a screening and diagnosis workflow, then validate the finalist on in-domain audio.

1. Use headline Average WER to narrow the field. Indian English now affects the main leaderboard average, so models that perform well only on read or non-Indian English are less insulated from a conversational Indian-accent test.
2. Check language coverage before comparing multilingual averages. The leaderboard source shows missing selected languages as NA and reports a Coverage value such as 3/5; a model receives a multilingual rank only when it supports all selected languages. See the integration code.
3. Slice the public Monsoon data by documented fields. Compare native district, state, city, device, or other available metadata. Look for spread and failure concentration, not only the best subgroup score.
4. Validate on representative production audio. Test your phone mix, noise conditions, code-switching patterns, VAD, and speech lengths. Public Monsoon clips average under ten seconds, so they cannot stand in for every call-center, meeting, or voice-agent workload.

On X, practitioner @PlimmApp warned that VAD can hide silence hallucinations when evaluation begins with speech-containing segments: “the vad step in your pipeline is also the blind spot… a model that emits fluent text over silence never gets charged for it.” Read the full comment. On-device teams should evaluate VAD, endpointing, and hallucination handling separately.

What Monsoon still cannot answer

• Scale and long-form transfer: The public en-IN set is 5.62 hours and the public hi-IN set is 1.33 hours; mean clips are 9.6 and 6.4 seconds respectively. Small score differences and long-audio behavior need separate validation.
• National representativeness: Broad district, state, and device coverage is not a population-weighted census of Indian English or Hindi speakers.
• Causal explanations: Regional associations do not identify whether the cause is training data, acoustics, phonetics, decoding, or another factor.
• Private-set predictiveness: Speaker-disjoint private splits are intended to test generalization, but their examples cannot be independently inspected.
• Downstream text fit: Search, subtitles, analytics, and legal transcripts may prefer different script, normalization, punctuation, or code-switching conventions.

Open ASR Leaderboard Monsoon FAQ

Does Indian English affect the main leaderboard average?

Yes. Monsoon en-IN is a default English-leaderboard input and contributes to Average WER. See the split inventory for its public and private sizes.

Is Hindi part of the main leaderboard ranking?

Hindi is shown in the multilingual view. Missing selected languages are displayed as NA, and only models with complete selected-language coverage receive a multilingual rank.

Can I download and score on the public sets?

Yes. The en-IN and hi-IN cards provide loading examples. Use corpus-level aggregation and OIWER for Hindi lattices.

Why is Hindi not scored with ordinary WER?

A fixed Hindi reference can penalize valid Devanagari, Romanized, code-mixed, or compound-word variants. The lattice records accepted forms for each transcript span.

Does a lower overall WER mean a model is better for every Indian region?

No. The official en-IN example shows nearly tied overall WER but different regional spreads. Inspect subgroup behavior and test representative audio before deployment.

The practical decision

Use en-IN for headline screening, hi-IN and OIWER for Hindi or multilingual comparison, and regional spread as a risk signal when the variation is large. Then validate the finalist on the audio, noise, VAD behavior, and text conventions your users actually produce.