Most voice recognition benchmarks are useless if you're building a real-world customer service bot. They're either English-only or use pristine studio recordings—nothing like the messy, multilingual chaos of actual support calls.
AI customer service company Sierra is trying to fix that with μ-Bench, an open-source evaluation dataset built from 250 real customer service calls and 4,270 human-labeled audio clips. All the audio is 8kHz mono, just like you'd get from a phone line. It's the kind of gritty, real-world data that actually matters when you're trying to deploy a voice agent that won't hang up on your customers.
https://twitter.com/btaylor/status/2046304347508920751
What's in the box
This public release is just a subset of Sierra's internal testing suite, which covers 42 languages, 79 regional variants, and over 13 different vendors. The open version includes results for five languages—English, Spanish, Turkish, Vietnamese, and Mandarin—and five major ASR providers: Deepgram Nova-3, Google Chirp-3, Microsoft Azure Speech, ElevenLabs Scribe v2, and OpenAI GPT-4o Mini Transcribe.
Sierra has released the code, the dataset (hosted on Hugging Face), and an open leaderboard, inviting other vendors to submit their own results.
The metric that actually matters
The most interesting part isn't the data—it's how Sierra measures success. They've introduced a new metric called UER (Utterance Error Rate), which separates errors that change the meaning of what was said from those that don't really matter.
Traditional Word Error Rate (WER) treats a missed "um" the same as a misheard phone number. But for a voice agent that's supposed to take action based on what it hears, only the second mistake will break the workflow. Two vendors with similar WER scores might have wildly different UER scores because they're making different kinds of mistakes.
What the numbers show
The results reveal some uncomfortable truths about today's voice AI landscape. Google's Chirp-3 leads in accuracy but is significantly slower at inference. Deepgram's Nova-3 is nearly 8 times faster in p50 latency but comes in last for multilingual accuracy.
Perhaps most telling: Mandarin recognition error rates can be up to five times higher than English, and Vietnamese performance varies dramatically between providers. These are the kinds of gaps you'd never see if you were only looking at English benchmarks.