What production-grade actually means, and how to tell if a vendor is bluffing

Production-grade means the system keeps working when the demo conditions stop holding, and you can prove what it did after the fact. Concretely, that reduces to four things a vendor either has or does not: observability, so you can see every call the system made; fallback, so a failing model degrades instead of taking the workflow down; versioning, so you know exactly which model and prompt produced a given output; and rollback, so a bad change can be reversed in minutes rather than rebuilt. If a vendor cannot show you all four, running against live traffic, you are looking at a prototype with good stage lighting.

The distinction matters because the demo is the easy part now. Foundation models are good enough that a competent engineer can build a convincing prototype for most enterprise workflows in a few weeks. Gartner predicted that at least 30 percent of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs, and unclear business value (Source: Gartner press release, 29 July 2024). Read the failure reasons rather than the headline. None of them are capability problems. They are the operational scaffolding that separates a demo from a system, and that scaffolding is exactly what "production-grade" names.

Observability: can you see what the system actually did

The first test is the cheapest to run and the most revealing. Ask the vendor to show you, for a specific request from last week, the full trace: which model version answered, what prompt and context went in, what the model returned before any post-processing, which tool calls fired, and how long each step took. A production system answers this in seconds because it logs every inference as a first-class event. A prototype answers it with a shrug, because the logs were built for debugging during development and thrown away in production.

Observability is not a dashboard of aggregate metrics. Aggregate uptime tells you nothing when a single wrong output lands in a clinical note or a compliance filing. What you need is per-request reconstruction: the ability to take one output someone is disputing and rebuild the exact conditions that produced it. In regulated deployments this is not a nice-to-have. When a hospital cannot reconstruct how a decision was reached, the accountability allocation in the contract becomes unenforceable, because nobody can establish which party's failure caused the outcome. Observability is the precondition for everyone else's liability being real.

There is a sharp buyer's question hiding here, and most incumbents fail it. Walk into a deployment and ask to see the last thirty times a human overrode the system. A vendor that captures overrides can show you the list, the timestamps, and what the model had proposed. A vendor that does not capture overrides will tell you the system rarely needs correcting, which is not an answer, it is a hope.

Fallback: what happens when the model is down or wrong

Foundation model providers are infrastructure now, and infrastructure has outages. Public status pages from the major LLM providers logged repeated multi-hour incidents through 2025 [verify]. Your system will call one of these providers on a day it is degraded. The only question is what your workflow does at that moment.

A production-grade system has a defined answer for three distinct failure modes, and you should make the vendor state each one. First, the provider is slow: does the system have a latency budget after which it times out and takes an alternate path, or does it hang and stall the queue behind it. Second, the provider is down: is there a secondary model or a degraded deterministic path, and has it actually been exercised, or is it a config file nobody has tested under load. Third, the model is up but wrong: what confidence or validation gate catches an output that is confidently incorrect before it reaches the user. That third case is the one prototypes never handle, because in a demo the model is never wrong.

Fallback is where the practitioner voice separates from the brochure. Anyone can say the system is resilient. Ask instead: when did you last trigger the fallback path in production, and what did the user see. A real answer includes a date and a description of graceful degradation. A missing answer means the fallback exists only in the architecture diagram.

Versioning and rollback: can you move and reverse safely

Models and prompts change constantly. A provider ships a new model version, you tune a system prompt, you swap a retrieval index. Each change alters behaviour, sometimes in ways that only surface on the traffic patterns you did not test. Versioning and rollback are the two controls that make change survivable.

Versioning means every output is stamped with the exact model version and prompt template that produced it, and those artifacts are stored, not just referenced. This is what lets you answer, three months later, why an output from March differs from the same input today. Without it, you cannot distinguish a model regression from a prompt change from a data change, and every incident becomes an archaeology project. Rollback means a bad change is reversed as a deployment operation measured in minutes, because the previous model and prompt versions are still deployable artifacts, not something you rebuild from memory.

The test is direct. Ask the vendor to describe the last time they rolled back a model or prompt change in production, how long it took, and how they detected the regression that triggered it. The shape of the answer tells you everything. A team that ships to regulated enterprises has a rehearsed story here with real numbers. A team that has only run demos will describe rollback as a thing they could do, in principle, if they needed to.

Where the change is a foundation model version rather than your own prompt, the discipline extends to the provider. When a new model version behaves differently on your traffic, you need to pin the old version, run both against a held-out set, and promote the new one only when it holds. Nextdot is registered in the Claude Partner Network, which is what lets that version-pinning and controlled promotion happen against the actual model release rather than being at the mercy of a silent upgrade.

How the four tests compose into one buyer's protocol

The four controls are not a checklist to tick separately. They compose. Observability is what lets you detect that a fallback fired or that a new version regressed. Versioning is what makes a rollback meaningful, because you know precisely what you are reverting to. Rollback is what makes versioning worth the storage cost. Fallback is what buys you the minutes to roll back without the workflow going dark. A vendor who has one but not the others has a partial system, and the gap is usually where the incident you have not had yet is waiting.

Run the protocol as one conversation. Pick a real request, ask for its full trace, ask what happens when the model is down, ask for the version stamp on the output, and ask for the last rollback story. Four questions, one deployment, ten minutes. A production-grade vendor answers all four with specifics and dates. A bluffing vendor answers with architecture, adjectives, and the promise that it would all work if you needed it to. The difference is not subtle once you know to ask, and it is far cheaper to find at procurement than at your first production incident.

Frequently asked questions