Why LLMs play the customer role badly

Executive summary. The idea “let one AI play the customer and another play our agent, and that’s how we test it” sounds logical but barely works. Language models are trained to be helpful, so in the customer role they start helping the agent within a couple of turns: answering to the point, not stumbling, not being difficult. The result is a greenhouse test, while a real customer breaks the agent at the first oddity. For the test to be honest, the simulator needs to be given specific models of difficult behavior. A library of “difficult people” is just as much an engineering asset as the agent’s code.

When a team builds automated testing for a dialog agent, a natural thought arises: seat a second AI across from the agent to play the customer. One agent sells, the other pretends to buy, the dialog runs on its own, tests scale. It’s a beautiful idea — and almost everyone trips over it, because a language model in the customer role behaves nothing like a customer.

Hypothesis: an LLM customer simulator is too reasonable by nature

Language models are trained for helpfulness: answer to the point, stay consistent, bring the conversation to a result. Those are exactly the qualities that make a bad test customer. Ask the model to “play an irritated buyer” — and it’s irritated for a turn or two, then collapses into cooperation: it starts phrasing the request clearly, answering clarifying questions, helping the agent reach a solution. It can’t hold the role, because the role contradicts how it was tuned.

Hence the hypothesis: a naive customer simulator doesn’t test the agent, it plays along. It produces a dialog that looks like a test but is closer in difficulty to an ideal scenario from a presentation than to a real conversation. This is a direct continuation of why synthetic dialogs are weaker than real ones: a generated customer is smooth and convenient where a live one is rough and unpredictable.

Problem: a greenhouse simulator gives a false green picture

The danger isn’t that the simulator is bad — it’s that it’s bad invisibly. There are many dialogs, they look varied, the pass rate is high, and all of this creates a sense of serious testing. In reality the agent was run against an opponent that helped it. The green report grows while real resilience stands still.

Then the familiar mechanism kicks in: management sees “the agent passed thousands of automated dialogs” and reads it as readiness. But thousands of cooperative dialogs aren’t worth one dialog with a customer who gets confused, gets angry, and changes the request three times. The agent ships to production, and the first genuinely difficult person finds what the simulator stepped around. This is still the same bias toward green, only now its source isn’t the development environment but the AI tester itself, which doesn’t know how to be difficult.

Why the usual approaches don’t work

The first reflex — “let’s ask the model to play harder” — gives a short effect and fades. A single general instruction “be a difficult customer” lasts the model a couple of turns; then the pull toward helpfulness outweighs it and it slides back into cooperation. Difficulty can’t be set with an adjective — it’s too vague to hold the role.

The second approach — “let’s take a more powerful model, it’ll play better” — misses. A stronger model is even more helpful and consistent, which is to say even worse as a difficult customer. The problem isn’t the model’s capability but the direction it was tuned in: to help, not to obstruct. Growing capability strengthens exactly the cooperativeness.

The third approach — “let’s add randomness, let it behave chaotically” — confuses noise with difficulty. Random lines make the dialog incoherent but don’t reproduce the specific ways real people break the agent: not abstract chaos, but anxiety, conflict, forgetfulness, suspicion. All three approaches share one root: we try to get a difficult customer without describing what the difficulty actually consists of.

Engineering model: a library of difficult-behavior models

The solution is to stop asking the model to “be difficult in general” and start giving it specific, narrow behavior models. Not “a difficult customer,” but a set of personas, each of which breaks the agent in its own way. In practice this is a library, and it matters more than the simulator model itself.

The anxious one demands repeated confirmations, re-asks the same thing, doesn’t trust the first answer — testing whether the agent can reassure without losing the thread. The conflict-prone one argues, nitpicks, provokes — testing whether the agent holds its line without sliding into either excuses or rudeness. The forgetful one loses context, contradicts itself, confuses its own data — testing whether the agent holds the dialog state on the customer’s behalf. The suspicious one looks for a catch, refuses to give data, doubts every offer — testing whether the agent can build trust. The impulsive one changes its mind on the fly, jumps around, rushes — testing goal stability. The bored one answers in one word, doesn’t engage, is ready to leave at any moment — testing whether the agent can hold attention. Each model is not an adjective but a specified behavior: what the persona does, what it avoids, how it reacts to the agent’s attempts to bring the dialog to a close.

Such a library plugs into a loop where the tester is not a human but another agent: the simulator takes one of the difficult-behavior models, runs a dialog aiming to obstruct the goal, and a separate judge evaluates the outcome — did the customer reach a result. The scenarios, meanwhile, are drawn not from imagination but from a corpus of real dialogs: live transcripts show which types of difficult people actually appear in your audience and in what proportion. The library isn’t invented in one sitting — it grows: every new way a real customer broke the agent becomes a new behavior model. Over time it turns into an asset — a description of exactly how your customers tend to be difficult, reusable for every new version of the agent. This approach is a special case of a red team for dialog systems: an independent loop whose job is not to help the agent but to prove it doesn’t work.

Practical takeaway for the business

Don’t take “the agent passed thousands of dialogs with an AI customer” as proof. Ask which difficult-behavior models were used and where they came from. If the simulator was simply told “be a customer,” the agent was tested against a helper, not a customer — and that test says nothing about production.

What to delegate. A library of difficult-behavior models is a separate artifact worth maintaining alongside the tests. To start, a few clearly described personas drawn from your real dialogs are enough; even a small set defined by behavior catches far more than a “universal difficult customer.” Where it matters doubly: wherever the agent works with emotionally loaded situations — sales, refunds, problem support — because that’s where anxiety, conflict, and suspicion appear most often.

What not to do. Don’t build testing on a single naive customer simulator — it will help the agent. Don’t scale up model power instead of describing behavior: a stronger model plays a difficult customer worse, not better. And don’t confuse randomness with difficulty — chaotic lines don’t reproduce the specific ways your agents break.

Build a library of difficult customers for your audience — talk to an engineer → .

Open questions

How many behavior models are enough depends on how diverse and emotionally loaded your audience is; the benchmark is covering the types that actually occur, not amassing a collection. How deeply a model can hold a difficult role at all is an open question: even with a clear behavior description, an LLM pulls toward cooperation, and that needs to be watched through the judge and the outcome metric. And where the line falls between “the agent was honestly tested on a difficult customer” and “the agent was forced into an unrealistically hostile dialog” — that’s set by the proportion of types from the real corpus, not by maximum harshness for its own sake. More broadly, this touches the limits of what AI knows about the user and the structural problems of multi-agent systems.

If your AI agent was tested by a single “universal” AI customer, it was almost certainly tested too gently. Discuss your case → — we’ll build a library of difficult customers for your audience and see where the agent breaks. Related work is in our cases and on the enterprise chatbots page.