Can Voice Agents Handle Bilingual Customers? Benchmarking Frontier ASR on Code-Switched Speech
ServiceNow AI researchers built a benchmark to test how well speech recognition systems transcribe code-switched speech, the everyday habit of bilingual people who swap languages mid-sentence. They ran seven frontier ASR systems across 918 synthetic utterances covering four language pairs. ElevenLabs Scribe V2 produced the best transcription accuracy, while OpenAI Whisper Large V3 Turbo finished last and often translated the speech instead of transcribing it.
Key Takeaways
- A Blog post by ServiceNow-AI on Hugging Face Back to Articles Can Voice Agents Handle Bilingual Customers?
- So, when a customer asked us how our voice agents would perform for their largely bilingual customer base who routinely code-switched, we decided to build our own benchmark and dataset to evaluate models.
We focused on automatic speech recognition (ASR) - the first step in any voice agent pipeline - because transcription errors propagate forward into every downstream component.
- To measure how various models perform, we report three metrics: Word Error Rate (WER), Semantic Word Error Rate (SWER), and Answer Error Rate (AER).
We choose these metrics to capture both (1) the models' exact accuracy in transcription, as well as (2) their ability to preserve the meaning of the utterance for downstream tasks.
- The Benchmark Data Pipeline We start with an internal corpus of IT support and HR interactions.
To create each code-switched utterance, we begin with parallel user utterances in English and one of our four non-English languages, then filter for good code-switching candidates.
- We then used an LLM verbalization pass to convert the text into its spoken form and used ElevenLabs Multilingual V2 to synthesize the audio.
A Blog post by ServiceNow-AI on Hugging Face Back to Articles Can Voice Agents Handle Bilingual Customers? Benchmarking Frontier ASR on Code-Switched Speech Community Article Published June 9, 2026 Upvote 42 +36 Shama Gupta shamagupta Follow ServiceNow-AI Lindsay Brin lindsaybrin Follow ServiceNow-AI Fanny Riols FannyRiols Follow ServiceNow-AI Introduction Over half of the world's population speaks more than one language. And for many bilingual speakers, code-switching - seamlessly switching between languages, even mid-sentence - is a natural part of everyday communication.
Whether in casual conversations, contact centers, or IT helpdesks, speakers fluidly adapt to whichever language feels most natural in the moment. Despite the prevalence of bilingual speakers across the world, there has been little work focused on how voice agents handle code-switched speech in enterprise settings. So, when a customer asked us how our voice agents would perform for their largely bilingual customer base who routinely code-switched, we decided to build our own benchmark and dataset to evaluate models.
We focused on automatic speech recognition (ASR) - the first step in any voice agent pipeline - because transcription errors propagate forward into every downstream component. In enterprise settings, where a misrouted ticket or misunderstood policy question has real operational consequences, getting the transcript right is an especially important step of the voice agent pipeline. Our benchmark covers four language pairs that were most relevant for our customer base: Spanish-English, French-English, Canadian French-English, and German-English.
For more details please read the original article at Hugging Face.
Continue Learning
Comments
Comments appear only after moderation. Your email identifies your submission to the moderator and is never displayed here.
No approved comments yet.