A red team for conversational AI

Executive summary. When a conversational AI agent is tested by the same team that built it, they aren’t testing the agent — they’re testing their own expectations, and they almost always get a green result. To learn the real reliability you need an independent loop interested not in confirming but in breaking it: a red team. In security this has long been the norm — nobody lets the developer audit their own work. For conversational AI it’s still rare, and that’s exactly why most assistants confidently pass the demo and fail with live customers. A red team moves the cost of that check to before launch instead of as a loss afterward.

Cybersecurity has a simple rule: a system you built cannot be objectively checked by whoever built it. That’s why penetration testers and red teams exist — people who are paid to break through your defences and prove they’re full of holes. Their success is your failure, found in advance. In conversational AI development this rule is almost absent, and that’s a mistake: a developer testing their own agent isn’t testing the agent — they’re testing their own picture of how the agent ought to behave.

Hypothesis: you can’t test your own agent honestly — you need an independent opponent

The thesis is simple and unpleasant. Any developer who sits down to check their own conversational agent unconsciously helps it. They phrase questions the way they themselves think about the task. They ask them in the order the system expects. They don’t confuse the agent, because there’s a “correct” scenario in their head and, without noticing, they lead the dialog exactly along it. As a result the test checks not the agent’s strength but the match between the agent’s behavior and the author’s expectations. And that’s always close to a hundred percent — otherwise the author would have already fixed it.

To measure real reliability you need a loop with the opposite motivation. Not someone who wants the agent to pass, but someone paid for the agent NOT to pass. Their job is to confuse the system, lead it astray, catch it in a contradiction, make it produce something wrong or dangerous. That is a red team, carried over from security to conversational AI. And increasingly it’s assembled not from people but from a second agent: “AI red team” is no longer a metaphor but an engineering discipline.

Problem: the developer tests their expectations, not the agent

Let’s look at exactly how a test’s honesty breaks down when the tester and the tested come from the same head.

First — phrasing. A person who knows the system asks “correctly”. They won’t write “can I get that thing, you know, for my wife, she’s in March, can’t remember the budget” — they’ll write “I want to choose a tour for two in March”. A real customer writes the first version, and that’s exactly what the agent trips over.

Second — order. The developer runs the dialog linearly: question, answer, clarification. A live person jumps around: starts on one thing, remembers another, returns to the first ten turns later. A test that follows the “correct” order will never reach the place where the agent loses the thread.

Third — absence of resistance. The author doesn’t argue with their agent, doesn’t lie to it, doesn’t suddenly change the premises. A real user does all of that constantly — not out of malice, but because human dialog is chaotic by nature. And when the development environment plays along too — fixes typos, picks the most likely interpretation, simplifies input — a green report becomes almost guaranteed. It grows while real resilience stands still.

This is the same illusion of green that falsifies quality at the level of the whole development environment: the system subconsciously helps itself pass the check, because no one in the loop is interested in the opposite.

Why the usual approaches don’t work

A team’s natural reaction to doubt about quality is to write more tests. It doesn’t help, and it matters to understand why.

More tests from the same people means more green under convenient conditions. Each new scenario is still written by a person who knows the “correct” branch and still leads down it. A hundred such tests check a hundred variants of the same cooperative behavior. None of them produces red under inconvenient conditions, because inconvenience isn’t built into the loop.

The second typical move is to hire a QA person and set up manual testing. Better, but it hits two walls. The wall of volume: a human will check dozens of dialogs, while you need thousands to catch the rare but expensive failures. The wall of non-determinism: the same dialog runs slightly differently with an LLM each time, so a single run proves nothing — you need statistics across many repeats of the same pressure, not a “checked” tick.

The third move is to hand testing to a naive LLM simulator. And here’s the trap that surprises teams: the simulator is too reasonable. Within a few turns it starts answering helpfully and helps the target agent bring the dialog to a close, because “bring it to a close” is exactly what it was trained for. A cooperative simulator reproduces the very same mistake as the developer: it checks how the system works when nothing gets in its way. And the whole point of a red team is to get in the way.

Here’s an anonymized example. A large travel retailer under NDA had a recommendation assistant that closed more than ninety percent of scenarios on internal tests. The team wrote the tests, and each ran down a clean branch: the customer names a destination, dates, budget — the agent recommends. When we set a real red team on it — an attacking agent with hard-behavior models and a corpus of real dialogs — the picture changed within the first evening. A customer who changed the dates three times in one dialog broke the agent’s context. A customer who named an unrealistic budget and then took offence at the lack of options drove the agent into apologies instead of recommendations. A customer who, mid-conversation about a cabin, suddenly asked about visas made the agent lose the original task. None of these failures was visible on the “correct” tests — because no developer talks to their own agent that way. Red was found exactly where the green loop never looked.

Engineering model: an opponent agent paid to fail

A red team for conversational AI isn’t a person with a checklist but a separate loop of three roles, designed to be interested in red.

At the center is the attacking agent — a red team executed by a machine. Its system task is phrased the inverse of the target agent’s: not to help the user, but to maximally obstruct the target agent from reaching the goal. It changes the topic mid-stream, slips in contradictory premises, returns to closed questions, gives wrong data and checks whether the agent notices, pushes with emotion, drags into adjacent scenarios. So it doesn’t slide into cooperativeness, it’s given not one role but a library of hard-behavior models: anxious, conflict-prone, forgetful, suspicious, impulsive, bored. This is an engineering artifact, not a temperament setting — each model reproduces a class of real people the agents break on.

The second role is the target agent, the very one that will go to customers. No concessions: the same stack, the same tools, the same constraints as in production. If the agent runs a lightweight version inside the red loop, the loop lies.

The third role is the judge. It assesses not the process but the outcome: did the dialog reach a real result for the user, did the agent emit anything dangerous, did it break on a contradiction. Crucially, the judge looks at the fact of the outcome, not at “did the agent call the right function” — the internal call can be correct while the customer still walked away empty-handed. This is a direct continuation of our engineering principle: the gate is the live outcome, not a green tick inside.

And the last thing, without which the whole loop loses meaning: the source of scenarios. Not synthetics from the developer’s head, but a corpus of real dialogs — actual chats, calls, support tickets. People break the agent in ways the engineer doesn’t imagine, and the only way to learn how is to take the test material from live conversations, not invent it.

Architecturally this is the same multi-agent task we solve in multi-agent systems: several agents with different goals in one loop, coordination, evaluation. So a red team naturally maps onto the competence of building multi-agent systems — it isn’t a new class of work but an application of it to QA. And precisely for that reason its typical pitfalls are the same as for multi-agent systems in general: role desync, context leaking between agents, a judge that quietly plays along.

Practical takeaway for the business

The main management conclusion is one: don’t trust the reliability assessment of an agent to those who built it. It’s not a question of trusting people — it’s a question of structure. The author objectively cannot check their own creation, however hard they try.

What to ask. Not “how many tests passed” but “who tried to break it, how, and what they found”. If the red team was played by the same development team, you don’t have an independent assessment — you have self-checking. If a naive simulator with no hard-behavior models did the testing, you’ve verified operation under ideal conditions that don’t exist in production.

What to set up. Independent testing of a conversational agent is a separate function, exactly like a security review — not a touch-up by the authors. To start, one attacking agent with a library of hard roles, a judge on outcomes, and a corpus of your real dialogs is enough. It’s cheap relative to the cost of an agent that confidently told a customer the wrong thing and cost you a deal or your reputation. If there’s nothing to build such a loop on, that’s a task for AI agent development, not for “tweaking the prompt”.

What not to do. Don’t release an AI agent into a customer-facing loop on a green report from its own developers. Don’t treat non-determinism as a detail — check across many runs of the same pressure, not one lucky one. And don’t confuse a human red team with a one-off “well, we poked at it by hand before release”: a single manual check doesn’t scale and isn’t reproducible, while what you need is a persistent loop that attacks on every iteration. Green CI is hygiene; proof that a real person reaches the result comes only from a red team.

Open questions

How aggressive a red team should be depends on the cost of error in the specific process. For a reference bot the bar is lower: the worst it does is fail to answer. For an agent that processes a payment or changes a booking, the red team must be far meaner, because the cost of a broken dialog is measured in the customer’s money.

Where the line lies between an honest attack and an unrealistic one is an open engineering question. Too weak an opponent finds no red; too absurd an opponent finds “red” that won’t happen in real life, and the team spends effort defending against the impossible. Tying the attack to a corpus of real dialogs helps calibrate it: the opponent is meaner than real people, but within their bounds.

And a management question that can’t be delegated to engineers: who decides, and by what criterion, that the agent has survived enough red to go to customers. That decision is made before launch, not after the first incident in production.

If you already have a conversational AI agent but it was “checked” by the same team that built it, you don’t yet have a real reliability assessment. Discuss your case → — we’ll assemble a red team and show which attacks your agent doesn’t pass yet.