RED-driven development: an AI agent's maturity lives in its red tests

Executive summary. “All tests green” doesn’t prove an AI agent works — it proves it was checked under convenient conditions. The maturity of an agentic system is measured not by how many tests passed but by the quality of the attacks it has already withstood. If no one deliberately tried to break it, you don’t have reliability data — you have hope. RED-driven development makes that check routine: you pay for it before launch instead of as a loss afterward.

Classic development trained us on the rhythm “build a feature — write a test — get green — move on”. For ordinary code that works, because code is deterministic: the same input gives the same output. An AI agent doesn’t behave that way, so the familiar green cycle creates false confidence instead of real confidence.

Hypothesis: for agents, maturity is the quality of RED, not the count of GREEN

A green test of a dialog agent says exactly one thing: in this one scripted scenario the agent reached the goal. It says nothing about what happens when the user stumbles, changes their mind, lies in the data, or returns to an old question twenty minutes later. Real reliability information comes not from green runs but from red ones — from situations where the agent broke, got caught, and was fixed. So maturity is measured by the quality and variety of reds the system has withstood, not the length of the green list.

Problem: chasing green falsifies quality

When a team measures progress by the number of tests passed, it starts optimizing that number — not real-world behavior. Scenarios are written by people who know the system and unconsciously steer the dialog down the “right” branch. Development environments amplify the bias: they simplify input, fix typos, pick the most likely interpretation. The green report grows while real resilience stands still. Worse, the green report gives management a false “done” signal, and the agent ships to production with untested edges.

Why the usual approaches don’t work

Classic TDD can’t be carried over to an agent, for two reasons. First, non-determinism: the same input gives an LLM different answers, so a single green run guarantees nothing; you need statistics across many runs, not a checkmark. Second, the tester’s cooperativeness: both a human and a naive LLM simulator start helping the agent within a few turns, because it’s easier to bring the dialog to a close. A test that plays along checks the author’s expectations, not the system’s strength. Adding more tests doesn’t help — more green under convenient conditions yields zero red under inconvenient ones.

The failure of agentic projects is systemic and predictable: it’s not about a “weak model” but about the absence of a loop that hunts for red. Where there is no such loop, the edge is found by the customer, not the team.

Engineering model: a cycle that produces red

RED-driven development changes the cycle itself. Instead of “feature → test → green” it looks like: build the agent → build an attacking agent whose job is to break it → get red → fix → get a new, harder red. The loop has three parts. A customer simulator runs a dialog aiming to maximally obstruct the goal — and is given hard-behavior models (anxious, conflict-prone, forgetful, suspicious), because a naive simulator starts helping far too quickly. The target agent is what we’re checking. A judge evaluates the outcome by fact: did we reach the goal or not, rather than “did the agent call the right function”. The source of scenarios is not synthetics but a corpus of real dialogs, because people break in ways a developer doesn’t imagine.

This failure taxonomy from real execution traces shows the key point: nearly 80% of failures are specification and coordination — that is, architecture, not model quality. RED-driven exists precisely to pull such failures into the open before production — where they show up as red, not as a lost customer.

Practical takeaway for the business

Measure maturity correctly. Don’t ask “how many tests passed” — ask “what attacks has the agent already survived, and who designed them”. If the answer is “the same team that built it”, you have no independent reliability assessment. Set up a separate loop interested in red: it’s cheap relative to the cost of an agent that confidently told a customer the wrong thing.

What to delegate. Independent agent testing is a separate function, like a security review — not a touch-up by the authors. To start, one attacking agent and a judge over a corpus of your real dialogs is enough; it pays off on the first batch of reds found.

What not to do. Don’t ship an agent into a customer-facing loop on a green report from its own developers. Don’t treat non-determinism as a detail: check across many runs, not one. Green CI is hygiene, not proof that a real person reaches the result.

Apply this to your agent — talk to an engineer → .

Open questions

How much red is enough depends on the cost of error in the specific process: for a reference bot the bar is lower; for an agent that processes a payment, much higher. How not to turn RED-driven into an endless war of agents — the judge and the corpus of real dialogs keep the loop tied to genuine, not invented, failures. Who decides “mature enough for production” — that’s a management question, settled before launch, not after the first incident.

If you already have an AI agent with green tests but no confidence it will withstand real people, that’s exactly the signal to set up a red loop. Discuss your case → — let’s see which attacks your agent hasn’t passed yet.