Skip to main content

Writing a multi-turn agent

A single-turn agent answers one input and returns one output — see Agents for that process(input) contract. A multi-turn agent holds a conversation: a simulated user talks to your agent across several turns, and the agent is scored on how the whole exchange goes.

This guide explains how Spec27 calls a multi-turn agent, how to keep conversation state between turns, how a conversation ends, and how to handle the edge cases that come up when you connect a real chat service. For why you'd run a multi-turn evaluation and how it's scored, see Goal-based multi-turn evaluation and Red-team multi-turn evaluation.

How a multi-turn run calls your agent​

A multi-turn run is a loop, and your agent is one side of it. The other side is a user simulator that plays the human. For each turn, Spec27:

  • Starts with the user. The conversation is always user-initiated — the simulator speaks first. Your agent never opens the conversation.
  • Calls process(input) once per turn, where input is that turn's user message text. One call returns one assistant turn.
  • Reuses the same runtime for the whole conversation. Each conversation gets its own runtime, and any module-level state you set survives from one turn to the next within that conversation.
  • Keeps going up to the max turns, or until the simulator ends the conversation (see Ending the conversation). The max-turns limit is set on the specification — for Goal Gold Team it defaults to 6 and can be set between 1 and 10.

The entry point is the same process(input) you use for a single-turn agent; what makes an agent multi-turn is that it remembers earlier turns.

Keeping state across turns​

Because the runtime is reused across turns, you keep conversation state in module-level variables — declared outside process — and read or update them on each call. The most common pattern is a running message history:

let messages = [];

export default async function process(input) {
  messages.push({ role: "user", content: input });

  const reply = await callYourService(messages);

  messages.push({ role: "assistant", content: reply });
  return reply;
}

The history starts empty for each conversation, then accumulates turn by turn, so your service sees the full context. Commit state only after a successful turn — update messages once you have a good reply, not before — so that if a turn fails, the earlier history is left intact.

The OpenRouter Multi-Turn Chat Completion Agent keeps a messages array exactly this way, and the OpenDialog Multi-Turn Agent keeps a session id across turns. Copy either from the registry as a starting point.

Ending the conversation​

The user simulator decides when the conversation ends, not your agent. It ends the run when it has reached the goal (or given up), and it always ends once the max-turns limit is hit. Your agent does not need to do anything special to finish.

Your agent can signal that it wants to stop — for example by replying with something like "I'll end the conversation here" — but that is just text. The simulator may or may not honor it. Treat it as a hint, not a guarantee.

Returning an empty reply (null, undefined, or "") is treated as a valid empty turn: it is recorded and the conversation continues. It does not stop the run.

Handling the edge cases​

Real chat services don't always behave like a clean request/response API. These are the cases to handle in your process function.

The service greets first​

Some services send an automatic greeting when a session opens ("Hi, how can I help?"). This greeting comes from the backing service itself — triggered by session initialization, not by any message from the user simulator. Your agent receives it before it has forwarded the simulator's input to the service.

Your agent should ignore it. Drain and discard the opening greeting, then return only the reply to the user simulator's actual input. Otherwise the greeting gets scored as your agent's answer to the first user message.

The service sends several messages per turn​

A turn is one process(input) call and one returned string. If your service replies with several messages for a single user message (for example a short acknowledgement followed by the real answer), Spec27 does not combine them for you.

Your agent should pool them. Wait until the service has finished sending messages for the turn, concatenate them into a single string, and return that. The OpenDialog Multi-Turn Agent does this — it joins the text of every message in the reply into one string.

The service errors or stalls​

If a turn fails, how you end it matters:

  • Returning nothing (an empty reply) is a normal empty turn — the conversation continues.
  • Throwing an error ends the conversation immediately. The turns that already happened are kept and the partial transcript is still passed to the judge, so the result reflects how far the conversation got before it broke.

So, when your service errors or returns nothing:

  • Retry inside process if it might help (for example a transient network failure). Retrying is your agent's responsibility — Spec27 does not retry a failed turn for you.
  • Throw when you cannot recover. A thrown error is the clean way to end the conversation; the partial transcript is preserved for scoring.
  • Set your own request timeout with an AbortController so a hung call surfaces as a normal error you can throw. If a turn hangs long enough to hit the sandbox timeout instead, the runtime is torn down and the conversation stops — you lose the chance to handle it yourself.

What's not yet supported​

These behaviors are not provided by the platform today. Where there's a workaround, it's noted above as something your agent does itself:

  • Automatic message pooling. Spec27 does not merge multiple service messages into one turn — your agent must pool and concatenate them (see above).
  • Platform retry of a failed turn. A thrown error ends the conversation; there is no automatic retry or resume of that turn. Retry inside process if you need it.
  • Recovering from a sandbox timeout. If a turn exceeds the sandbox timeout, the runtime is torn down and the conversation stops. Use an in-agent timeout to stay in control.