Voice AI unicorn ElevenLabs has open-sourced its real-time voice conversation component, Speech Engine Skill. It follows the Agent Skills open specification, designed to let AI agents and LLM apps quickly integrate high-fidelity, low-latency voice interactions. Developers just need to run npx skills add elevenlabs/skills to add the voice engine to their project runtime — no need to wrangle multiple APIs or build a complex state machine.
https://twitter.com/ElevenLabsDevs/status/2059646861913330031
Speech Engine Skill is built on high-performance WebSocket connections, each representing a call session. When a user speaks, the browser captures audio and streams it to ElevenLabs, which transcribes it in real time and pushes the text to the developer's server. The server generates a streaming text response via an LLM, sending it back using the SDK's sendResponse() or send_response() function (which accepts a string or an async iterator). ElevenLabs then converts that response into low-latency synthetic speech that plays in the browser. The SDK handles network routing, request signature verification, heartbeat monitoring, and session lifecycle management under the hood, with native support for interruption and turn-taking.
To simplify front-end development, ElevenLabs is simultaneously releasing @elevenlabs/react and @elevenlabs/client libraries. With minimal code and a server-issued secure session token, a front-end page can quickly spin up a digital voice assistant that's resistant to noise and interruptions. For production deployments, ElevenLabs recommends treating the transcribed text as untrusted input and setting up deterministic guardrails or intent allowlists on the server side — don't let raw speech-to-text output directly map to privileged LLM actions or sensitive tool calls.