Latency in an AI voice agent is the sum of five slices: telephony, speech-to-text, the language model, the data fetch from hospital systems, and text-to-speech. In hospital deployments the data fetch is usually the largest and least controllable slice, and concurrency at peak hours widens every slice at once. You reduce latency by measuring each slice separately, then streaming, pre-fetching, caching and choosing a smaller model where the task allows it.
The budget a caller will tolerate
Start with the caller, because the caller sets the budget. Across ten languages studied, Stivers and colleagues found that the most common gap between one speaker finishing and the next starting sits between 0 and 200 milliseconds (Stivers et al., PNAS, June 2009).
Telephony engineers have worked with a similar ceiling for decades. ITU-T Recommendation G.114 treats one-way transmission delay below 150 milliseconds as essentially transparent for speech, and delay above 400 milliseconds as unacceptable for general network planning (ITU-T G.114, May 2003). That covers the network alone. A voice agent adds listening, thinking, lookup and speech on top.
That stack puts a voice agent well past a human's turn gap. The practical question for a hospital is how far past it the agent runs, and where the extra time goes. Twilio's guide to core latency in AI voice agents sets a median target of about 1.1 seconds from the caller finishing to the agent's voice reaching the caller's ear, with 1.4 seconds as the upper limit (Twilio, November 2025). Your own callers and systems set the real number.
In Nextdot's live voice deployments, latency is the second most common failure, after human handoff. It comes from data fetch, and at some sites from scale. Both are covered below.
The five slices
Break the response budget into slices and put a timer on each one. Log the 95th percentile per slice as well as the median.
Telephony. Audio travels from the caller's handset through the carrier, into the hospital's SIP trunk and onto the media server. At Nextdot that server is FreeSWITCH. Codec choice, jitter buffers and the physical distance between the SIP trunk and the servers running the agent all add time. This slice is mostly fixed once the architecture is chosen.
Speech-to-text and endpointing. The agent has to decide the caller has finished speaking before it can answer. That decision, called endpointing, is a trade-off. Set the silence threshold short and the agent interrupts elderly callers who pause mid-sentence. Set it long and every turn starts with dead air. Callers who switch between Hindi and English mid-sentence can slow a recogniser before the transcript is final.
The language model. Two numbers matter: time to first token and tokens per second. Time to first token grows with prompt length, so a system prompt stuffed with every policy the hospital owns costs milliseconds on every single turn.
The data fetch. The agent needs the doctor's schedule, the patient's booking, the department's timings. That means a call into the HMIS or an appointment system, usually through an API the vendor built for a web front end with a human user.
Text-to-speech. Synthesis has its own time to first audio.
Why data fetch is the slice you do not control
The other four slices sit inside the voice stack, where the engineering team can tune them. The data fetch sits inside someone else's system.
Hospital information systems were built for clerks at terminals who can wait a couple of seconds for a screen to load. A slot availability query that feels instant to a front-desk user can be slow enough to break a phone conversation. Some HMIS deployments sit on-premises behind a VPN that adds its own round trip. Some slow down during the morning OPD rush, exactly when call volume peaks.
A better model or a faster speech engine leaves this slice untouched. The fix sits in the integration layer: a read path designed for voice, with the data the agent needs shaped and staged before the caller asks. The architecture Nextdot builds toward is adaptable integration over the existing HMIS, through MCP and open APIs, with the HMIS left in place. "Your Integration Layer Decides Whether AI Works" covers the integration layer in general. For latency, the point is narrower: the slowest query in your HMIS sets the floor for your voice agent's response time unless you design around it.
Choosing the model inside the budget
Most turns in a hospital contact centre are routine: which days does the cardiologist sit, is my appointment confirmed, what time does the lab open. These turns need the right fact, fetched quickly and said clearly. They rarely need the reasoning depth of a frontier model, and a larger model spends milliseconds the budget does not have.
Nextdot is registered in the Claude Partner Network, and our voice stack runs Claude Haiku, a smaller Claude model, for this reason: inside a voice budget, time to first token outweighs the extra reasoning depth that a larger model brings to a turn that is mostly retrieval. The stack is designed by a team that includes two engineers holding the Anthropic Claude Certified Architect (CCA-F) credential, and the model choice follows from the budget.
If you run more than one model, write the routing rule down. A sound rule matches each intent to the smallest model that holds the quality bar for it, and gets re-checked whenever the intent mix or the model changes. Turns the agent should not handle go to a person, through the escalation design covered in "The Handoff Is the Product".
Four techniques that move the budget
Streaming. Run every stage as a stream. The recogniser sends partial transcripts, the model starts generating as soon as the transcript is final, and synthesis begins on the first complete clause instead of waiting for the full answer. Streaming leaves each slice the same size and overlaps them, so the caller hears the first words while the rest of the sentence is still being produced.
Pre-fetch. Most hospital calls follow a small number of paths. When the caller's number matches a registered patient, the agent can pull upcoming appointments before the caller states the reason for calling. When the caller names a department, the agent can request that department's schedule while it is still confirming the name. Pre-fetch moves the data fetch off the critical path. It also touches patient data before the caller has asked for it. The hospital is the Data Fiduciary under DPDP 2023, so take two questions to its counsel before switching pre-fetch on: does fetching ahead of the request fit the purpose the data was collected for, and how long may pre-fetched data be held?
Filler. A short, honest acknowledgement such as "Let me check the cardiologist's schedule" buys a second or two while the fetch completes. Used on every turn, it reads as stalling. Keep filler for turns that genuinely wait on a system, and never use it to cover a fetch that has already failed. A failed fetch needs a recovery path or a handoff.
Caching. Doctor rosters, department timings, hospital addresses and general visitor information change slowly. Cache them close to the agent with a short, explicit expiry, and refresh on a schedule instead of on every call. Slot availability changes minute to minute and should stay live. The design question is which data can be stale for five minutes without harming a patient, and the hospital, as the owner of that data, has to sign off on the answer.
What scale does to the budget
A voice agent that answers in under a second with ten concurrent calls can run much slower with two hundred. The dependencies around the model usually break first. Model API rate limits start to bite, the HMIS connection pool saturates, the media server's CPU climbs as it handles more audio streams, and queued requests stack their waiting time onto every slice. "Why Your AI Project Succeeded in the Pilot and Failed at Scale" covers this pattern in general. On a phone line the caller hears every extra millisecond as silence.
Plan for scale at three points. Load test at the hospital's real peak concurrency, often the morning appointment rush, and measure per-slice latency at that load. Protect the HMIS with a read replica or a staged cache so the voice agent's traffic cannot slow down the front desk. Set a timeout per slice, with a defined fallback when it expires: a cached answer, a callback offer, or a transfer to a human.
What to ask for before go-live
A CIO evaluating any voice agent should ask for four things. A per-slice latency breakdown, with median and 95th percentile, measured on the hospital's own systems. A load test result at the hospital's peak concurrency. The timeout and fallback for each slice. And the logs that let the team reconstruct any slow call after the fact. Observability runs on every Nextdot deployment, with audit logs and versioned models and prompts, so the slow call a caller complains about can be traced to the slice that caused it.
A vendor who can quote only a single average response time has not measured where the latency goes.
Frequently asked questions
What is acceptable latency for a voice agent?
There is no single regulatory standard. In human conversation the most common gap between turns is 0 to 200 milliseconds (Stivers et al., PNAS, June 2009), and ITU-T G.114 treats one-way network delay above 400 milliseconds as unacceptable for general network planning (May 2003). For AI voice agents, Twilio's November 2025 latency guide targets a median of about 1.1 seconds from the caller finishing to the agent's voice reaching them, with 1.4 seconds as the upper limit.
Why do voice agents pause before answering?
The pause is the sum of five slices: telephony, speech recognition and endpointing, the language model, the data fetch and speech synthesis. In hospital deployments the longest part is usually the lookup into the HMIS or appointment system.
How does data fetch affect voice agent speed?
The data fetch is often the largest slice and the one the voice stack controls least, because it depends on the speed of the hospital's own systems. HMIS platforms built for front-desk screens can respond too slowly for a phone conversation, especially at peak hours. Pre-fetching likely data, caching slow-changing information and building a dedicated read path for the agent all reduce its impact.
Does latency get worse at scale?
Yes, if the system was not designed and tested for peak load. As concurrent calls rise, model rate limits, HMIS connection limits and media server capacity all add waiting time to each slice. Load testing at real peak concurrency, protecting the HMIS with a replica or cache, and setting a timeout and fallback for each slice help hold response times as volume grows.