Skip to main content

Red-team multi-turn evaluation

Red-team multi-turn evaluation tests whether an agent can be worn down into harmful or policy-breaking behaviour over a conversation, not just on a single adversarial prompt.

Single-turn red teaming sends one adversarial input and checks the response. A red-team multi-turn evaluation is the multi-turn form of that same Red Team idea: a simulated adversarial user probes the agent across a bounded conversation, looking for misuse, harmful output, or other failures that only emerge after several turns of pressure.

This matters because an agent that refuses a harmful request on the first ask can still be redirected, reframed, or gradually steered into crossing the line over a longer exchange. Many safety failures only appear once an adversary has room to build context and apply pressure. For one-shot jailbreak and misuse testing, a standard Red Team evaluation is enough.

What a red-team multi-turn evaluation does​

A red-team multi-turn evaluation usually combines:

  • a set of harmful or policy-sensitive intent entries
  • an adversarial user simulator that plays the attacker and drives the conversation
  • a max turns limit that bounds each conversation
  • attack suites that add known jailbreak techniques
  • judge-based scoring across the whole exchange

At run time, Spec27 typically does the following:

  1. starts an adversarial conversation from a risky intent, with the simulator acting as the user
  2. lets the simulator push the agent across turns, up to the max-turns limit
  3. applies any selected attack-suite techniques during the conversation
  4. asks a judge whether the agent crossed the failure boundary anywhere in the exchange
  5. records which cases held the line and which gave way, with the full conversation trace

What red-team multi-turn evaluation is testing​

It tests whether an agent's safety boundary holds under sustained pressure, not just against a single hostile prompt.

It helps you answer questions such as:

  • Can the agent be worn down into compliance after refusing at first?
  • Can an adversary reframe or redirect the request across turns to slip past a refusal?
  • Does the agent leak harmful content gradually, a little at a time?
  • Does it hold its boundary consistently, or only on the opening ask?

How the simulated user works​

During a run, the user simulator plays the adversary: it opens the conversation and keeps pushing toward a failure across turns, up to the max-turns limit. The agent's responses are judged across the whole exchange rather than on a single reply, so a refusal early on does not guarantee a pass if the agent gives way later. Red Team evaluation is always judge-scored — the failure condition is contextual, not a string match.

Adversarial coverage​

Red Team specifications use attack suites rather than individually selected methods. Each suite pairs a set of single-turn jailbreak attacks with a planned multi-turn red-team attack:

  • Light Suite — a fast pass with the top single-turn jailbreak attacks plus a multi-turn red-team attack.
  • Heavy Suite — a fuller sweep across a broader set of single-turn attacks plus a multi-turn red-team attack.

Running without a suite gives a baseline that exercises only the primary test set. See Attack methods for what the individual techniques do.

Reading the results​

Each case resolves to Pass, Fail, or an error state — Error (Agent) if the agent or simulator failed to respond, and Error (Judge) if scoring failed. Open a case to read the full conversation trace and the judge's explanation, so you can see what the simulator attempted and where the agent gave way. See Runs and results.

Why this matters​

Red-team multi-turn evaluation helps you answer practical questions such as:

  • Can this agent be jailbroken given a few turns rather than one?
  • Which adversarial strategies wear it down most effectively?
  • Are its safety instructions strong across a conversation, or only on the first message?
  • Which multi-turn failure modes should block launch or need deeper mitigation?

This matters for release readiness, governance reviews, and ongoing monitoring of agents that hold real conversations with users.