ElevenLabs Releases Two Real-Time Speech Conversion Models

Speech AI company ElevenLabs has released two variants of "Eleven Realtime," a new speech conversion model specialized for real-time conversations. The model adopts technology called "speech-to-speech," which converts voice directly into different voice without converting it to text first, with the primary aim of enabling real-time conversations with AI.

Speech-to-speech is a technology that processes audio directly from speech to speech without first converting voice input into text. Traditional speech AI systems typically go through multiple steps: "speech→text→text processing→speech," which inevitably introduces latency. The approach ElevenLabs has chosen this time is positioned as one that pursues faster response by skipping these conversion steps.

Among the two models released this time, the top-tier "v1 Max" variant achieved a 98.8% score on the Big Bench Audio benchmark, a performance evaluation metric for speech processing models. The response time is reported as 0.57 seconds, and ElevenLabs announced that no model currently exists on the Artificial Analysis leaderboard that simultaneously exceeds this speed and accuracy.

The significance of these numbers lies in performance directly linked to the "naturalness" of voice conversations with AI. When people talk to each other, a gap of about 0.5 seconds is barely noticeable, but beyond that, unnaturalness becomes apparent. A response time of 0.57 seconds is close to that threshold and demonstrates the potential for users to engage in conversation without discomfort.

Regarding real-time voice AI, various applications are being considered, including customer support, language learning, and medical consultation. For these uses, achieving both "accuracy" and "speed" simultaneously is essential, and lack of either makes practical use difficult. ElevenLabs' emphasis on this dual achievement in this announcement can be seen as a message-driven approach mindful of these market requirements.

In the speech AI field, multiple companies have been advancing development toward improved accuracy and lower latency over the past several years, with competition continuing. ElevenLabs is originally known for speech synthesis (technology that generates speech from text), but the introduction of this real-time speech conversion model can be seen as a move to strengthen its presence in the conversational AI domain. Going forward, the quality of the model in actual service environments and comparative verification with competitors' models will be key points of interest.