Testing a voice agent before patients hear it

You test an AI voice agent before go-live with six things: a scripted call set for every intent it handles, a test set built from real accents and code-switched speech, a set of adversarial and distressed callers, a load test at the hospital's actual peak concurrency, a regression suite that reruns whenever a prompt or model changes, and written go-live criteria agreed before the first test runs. A demo call from the vendor's office proves the agent can talk. It says nothing about the four hundredth call on a Monday morning.

A voice agent can pass its tests and still disappoint in production, because the tests were the wrong shape. A handful of clean calls in quiet rooms, spoken by the people who built the prompt, measures the agent against its own authors. Patients bring a different distribution: bad lines, elderly callers who pause mid-sentence, Hindi and English inside one sentence, and anxiety that shows up as interruption. The work before go-live is building a test that looks like that distribution.

Scripted call sets, one per intent

Start from the intended-use statement. Every intent the agent is allowed to handle gets its own call set: book an OPD appointment, reschedule, cancel, ask a doctor's timings, ask for directions, ask about a report's status. Every intent the statement excludes gets a set too, because the agent's job on those calls is to recognise them early and route them.

A useful call set for one intent covers the happy path, then the variations that break slot-filling. The caller gives the doctor's first name only. The caller names a department the hospital calls something else. The caller asks for "next Tuesday" on a Sunday night. The caller changes their mind halfway through the booking. The caller's registered number does not match the name they give. That last case matters more than it looks: an agent that books against the wrong record has produced a system-behaviour failure, and in Nextdot's view system behaviour sits with the vendor.

Each scripted call needs a pass condition written down before it runs. "The agent handled it well" is an opinion. "The agent booked Dr. X on the correct date, read the slot back, and the booking appears in the HMIS" is a test. The pass condition for an out-of-scope call is a correct transfer with the context attached, which ties directly to the handoff design in "The Handoff Is the Product".

Accent and code-switch test sets

Speech recognition error does not fall evenly across speakers. Koenecke and colleagues tested five commercial speech recognition systems in the US and found an average word error rate of 0.35 for Black speakers against 0.19 for white speakers (Source: Koenecke et al., PNAS 117(14), 7 April 2020). The authors traced most of the gap to the systems' acoustic models, the part that turns sound into words. The same risk applies to a hospital line in India, where the Census of India 2011 counted 26 percent of the population as bilingual (Source: Census of India 2011, Table C-17, Office of the Registrar General), and where a caller moves between languages inside a single sentence.

So the accent set should be recorded by people who sound like the hospital's callers: the regional accents of its catchment, older voices, callers on speakerphone, callers on a two-wheeler at a signal. The code-switch set should hold sentences the way patients actually say them, with an English drug name inside a Hindi question, or a Hindi date inside an English complaint. Score each clip on word error rate and, more usefully, on whether the agent extracted the right slot. A transcript can be imperfect and still carry the correct doctor and date.

This set is also where the speech-to-text choice gets tested honestly. Nextdot is part of Sarvam's Startup Partner Program, and the reason that matters here is narrow: a recogniser tuned on Indian speech is the component these sets are designed to stress, and a model tuned mostly on other speech is the one most likely to struggle on the clips a hospital line produces most.

Adversarial and distressed callers

Two groups of callers need their own test sets, and both are uncomfortable to write.

The first is the adversarial caller. Someone asks the agent for another patient's appointment details. Someone tries to talk the agent out of its instructions. Someone asks a medical question and presses for a dosage. Someone repeats "human" and nothing else. Pass conditions here are refusals and transfers, and each one should be checked against what the agent said aloud, since a refusal that leaks the other patient's name has already failed.

The second is the distressed caller: chest pain, a fall, a child who will not wake, a caller who is crying. These calls test whether the agent stops whatever it was doing and routes to the urgent path the hospital has named. Script them with the hospital's clinical and patient-access leads, because the right route differs by site, and escalation paths are organisation-specific. There is no single escalation model that suits every hospital. The test set only needs to prove the route fires every time the trigger appears.

Load tests at peak concurrency

An agent that answers promptly at ten concurrent calls can slow sharply at two hundred, and the model is rarely what gives way first. "Where the Latency Goes" breaks down where the time goes. For testing purposes the rule is simple: find the hospital's real peak, which is often the morning appointment rush, and run synthetic calls at that concurrency against the hospital's actual HMIS integration, on a replica or a staging system that behaves like production.

Measure three things at peak. Per-turn latency at the median and the 95th percentile. The share of calls where a timeout triggered a fallback, and whether that fallback was correct. And the effect of the agent's traffic on the hospital's own systems, because a voice agent that slows down the front desk has moved the problem to a different queue. Run the load test with realistic call mixes, including transfers, since a transfer holds a line open while a human picks up.

Regression tests when prompts or models change

A voice agent in production keeps changing. The hospital adds a department, a prompt gets a new instruction, a model version updates, a speech engine changes its endpointing. Each change can fix one intent and quietly break another.

The regression suite is the scripted sets, the accent and code-switch sets and the adversarial sets, frozen with their pass conditions and rerun automatically against every change before it ships. The comparison is always against the last passing version, intent by intent. A change that improves booking accuracy by a point and breaks the chest-pain route has failed, whatever its average says.

This only works if every change is versioned. Observability runs on every Nextdot deployment: audit logs, versioned models and prompts, and human override capture. That versioning is what makes a regression test meaningful, because it tells you exactly which change the suite is judging. Formal evaluation against clinician-labelled ground truth does not yet run on every Nextdot account, and extending it is a current engineering priority. Before go-live, a buyer should ask any vendor, including Nextdot, to show the test suite that will run on their deployment, what it contains, and how often it reruns.

Go-live criteria, agreed before testing

Go-live criteria are the contract between the hospital and the vendor about what "ready" means. Write them before the first test, sign them on both sides, and do not move them because the launch date is close.

A workable set for an inbound hospital line covers five areas. Intent accuracy: a pass rate per intent on the scripted sets, with a higher bar for booking and cancellation than for directions. Recognition: a slot-extraction pass rate on the accent and code-switch sets that the hospital's patient-access team considers acceptable for its callers. Safety: every distressed and adversarial test passes, with zero tolerance, since one wrong route on a chest-pain call outweighs a thousand good bookings. Load: latency and fallback behaviour at peak concurrency within the agreed budget. Handoff: every transfer arrives with context, and the human never has to re-ask for what the agent already captured.

Then go live small. One line, one department, or one time window, with humans listening to a sample of calls every day. In Nextdot's live voice deployments, human handoff is the most common failure and latency, from data fetch and at some sites from scale, is the second. A staged launch is where the test set meets the real distribution, and every failure found there becomes a new test case in the regression suite.

Every Nextdot voice deployment today is inbound. Outbound calling, reminders and follow-ups are separate designs with separate consent questions, and the test plan above does not cover them.

Frequently asked questions