What we mean by production

The demo problem

A multi-agent system demos well. You give it a task, the agents coordinate, and something useful comes out. Everyone in the room can see it working.

What the demo doesn't show is the run where agent seven times out at ninety seconds while agents one through six have already committed their work, agent eight is waiting on seven's output, and the whole thing is now holding open a connection nobody will close.

Getting from the first to the second is most of the engineering.

Six things that break first

1. Partial failure

Fourteen agents running in parallel means fourteen independent chances to fail per task. At any realistic per-agent failure rate, tasks where everything succeeds become the minority quickly.

So the orchestration has to answer: what does the system do when three of fourteen fail? Retry just those three, fail the whole task, or return a partial result and mark it?

The answer we arrived at was to make it explicit per agent. Some are required and their failure fails the task. Others are enriching and their failure produces a result with a flag on it. That decision belongs in configuration rather than in code, because it changes per use case, and it needs making before launch rather than during an incident.

2. Malformed output

Models return text. You asked for JSON. Most of the time you get JSON.

The failure isn't dramatic. It's a trailing comma, a code fence around the object, a field that's a string this time and an array last time, or a perfectly valid object with a hallucinated extra key. Any of these will break a strict parser.

Validate every output against a schema at the boundary, before it enters the rest of the system. Where validation fails, retry once with the error appended to the prompt, which resolves a surprising share of cases. Where it fails twice, treat it as an agent failure and route it through the partial-failure path above. What you must not do is coerce a bad response into a usable shape and carry on, because the wrongness then travels.

3. Retry storms

The obvious response to a failure is a retry. With fourteen agents, the obvious response is also how you take down your own provider quota.

Retries need exponential backoff with jitter, a hard attempt limit, and a circuit breaker at the provider level. When a provider is degraded, every agent fails at once, every agent retries at once, and without a breaker you convert a slow API into a total outage of your own making.

4. Cost

A system that costs a reasonable amount per task at ten tasks a day can be alarming at a thousand, and the mechanism is rarely volume alone. It's usually a retry loop, a prompt that grew, or a context window that now includes a conversation history nobody trimmed.

Three things are worth having from the first day. A per-task cost ceiling that fails the task rather than exceeding it. Token counting logged per agent, so you can see which one grew. And an alert on daily spend, not monthly, because monthly tells you after it happened.

5. Observability

When a task produces a wrong answer, you need to know which of fourteen agents was wrong, what it received, and what it returned.

That means every agent call is logged with its inputs, outputs, token counts, latency and outcome, correlated by a task ID that runs through the whole chain. Without the correlation ID this is impossible in practice, and adding it later means touching every call site.

The test is whether you can answer "why did task 8812 produce that" in under five minutes, using only what's already stored. If the answer involves re-running anything, you don't have observability, you have logs.

6. Latency under parallelism

Fourteen agents in parallel is only fast if they're genuinely parallel. Shared rate limits, a connection pool that's too small, or a synchronous step in the middle will serialise them quietly.

The symptom is that per-agent latency looks fine in isolation and the end-to-end time is four times what the arithmetic says it should be. Measure end to end, not per call, or you won't see it.

What this means for evaluating a supplier

If you're deciding whether somebody has taken this kind of system to production, ask them about partial failure and cost ceilings. Both are unavoidable in a real deployment and neither comes up in a proof of concept.

Somebody who has run one of these will have an opinion about circuit breakers and will tell you about a bill that surprised them. Somebody who hasn't will talk about model selection and prompt engineering, which are the interesting parts and not the parts that break.