There is a huge gap between how AI voice agents sound in demos and how they perform in production in the first month. Not because the technology does not work — it does, and it works well — but because the problems that appear in production are exactly the ones nobody mentions in the demo.
This guide is for anyone considering implementing a voice agent in their company and wants to know what they’re really up against.
The problem of accent and colloquial language
Speech-to-text models are trained primarily on large audio corpora in English and, to a lesser extent, in standard Spanish. The result is that they work very well with neutral pronunciations and less well with strong regional accents.
Basque as a language is not the problem — the agent can be configured to understand and respond in Basque perfectly well. The problem is Spanish spoken with a strong Bilbao accent, the use of Basque words within Spanish sentences, and the local colloquial expressions that any Bilbao resident uses naturally in an everyday call.
It is not an insurmountable problem. It is resolved with a combination of adjusting the speech recognition model, configuring a custom vocabulary with the most frequent terms in the business, and designing the conversation flows so that the agent asks for confirmation when the transcription does not have sufficient confidence. But it requires work and real-call production tests before the system runs smoothly.
The latency you feel and the latency you don’t
The latency threshold below which a pause in the agent’s response feels natural —not artificial— is around eight hundred milliseconds. Above one second, the other person starts to notice that something is not quite human.
Achieving that latency consistently requires all three system components — STT, LLM, TTS — to be on the same infrastructure or geographically close. Using one provider’s STT, another’s LLM, and a third’s TTS with network calls between them multiplies latency.
On a VPS in Frankfurt with local LLaMA, Deepgram for STT and ElevenLabs for TTS, the average latency is between six hundred and nine hundred milliseconds under normal load conditions. During peak usage or with complex queries that require more model processing, it can exceed one second.
The system design must account for these cases: what the agent does when processing takes longer than expected, how it signals to the caller that it is processing without the pause feeling awkward.
The case nobody anticipates: the angry customer
Designing conversation flows for polite customers who call with a clear query is relatively straightforward. Designing flows for angry customers is significantly harder and is rarely done well in the first implementation.
A customer who calls to complain that their order arrived late doesn’t want to speak to a voice agent. They want to speak to a person who will listen to them, understand their frustration and take some concrete action. If the agent tries to process that call as though it were a standard query, the experience is actively bad.
The system must be able to detect the emotional tone of the call — something that modern language models do reasonably well — and escalate immediately to a human agent when it detects high frustration, regardless of the content of the query.
The handover of the call to the human agent must include the full context of everything said up to that point, so the human agent does not start from scratch and the customer does not have to repeat their issue.
The maintenance that never stops
A voice agent in production is not a project with an end date. It is a system that requires continuous maintenance.
Every week there are calls with queries the agent did not handle correctly. You need to review those cases, understand why they failed, and update the conversation flows or the agent’s knowledge base so it works better next time.
Prices change. Schedules change. Available services change. Every change in the business has to be updated in the agent’s configuration.
The voice model that sounds best today may have an improved version in three months. Updating the model may require adjustments to speech rate, pause management, and synthesis parameters.
Anyone implementing a voice agent must be clear that the first month in production is an intensive adjustment period. Results in month three are significantly better than in month one, and those in month six are better than those in month three. Continuous improvement is part of the process.
When it makes sense and when it doesn’t
A voice agent makes sense when a business loses calls because it cannot handle them all — outside opening hours, during demand peaks, with the team occupied in service — and when the calls being missed are mostly information requests or appointment requests that the agent can resolve well.
It makes no sense when the business value lies in the human, personalised relationship that the team provides — a specialist consultancy, a high-end service — because the voice agent will always convey less warmth than a person who knows the client.
The most useful criterion for making the decision is this: how many of the calls you receive each week could a new employee handle perfectly well with two weeks of training, and no specialist knowledge? If the answer is more than fifty per cent, the voice agent makes sense.