Skip to main content

Goal-based multi-turn evaluation

Goal-based multi-turn evaluation tests whether an agent can complete a task over a back-and-forth conversation, not just answer a single prompt.

Most Gold Team evaluation is single-turn: one input goes to the agent, and one output comes back to be scored. A Goal Gold Team evaluation is the multi-turn form of that same Gold Team idea — it measures whether an agent reaches a desired outcome when the work unfolds across several exchanges. A simulated user drives the conversation toward a defined goal, and the agent is scored on whether the goal is reached rather than on any one reply.

This matters because many real tasks are conversations, not single questions. Booking something, gathering required details, troubleshooting, or guiding a user through a flow all depend on what the agent does turn after turn — asking the right follow-ups, holding context, and converging on the outcome. If a single prompt and response is enough to judge the behaviour, a standard Gold Team evaluation is the simpler choice.

What a goal-based evaluation does​

A goal-based evaluation usually combines:

  • a Goal for each case — what the simulated user is trying to achieve
  • a Target for each case — the outcome that counts as success
  • a user simulator that plays the user and drives the conversation
  • a max turns limit that bounds how long each conversation can run
  • a multi-turn capable agent under test

At run time, Spec27 typically does the following:

  1. starts a conversation from the case's goal, with the simulator acting as the user
  2. lets the agent and the simulated user exchange turns, up to the max-turns limit
  3. follows the conversation as it works toward the target outcome
  4. judges whether the goal was reached across the whole exchange
  5. records each case as a pass or a failure, with the full conversation trace

You set the goal and target entries, the user simulator, and the max-turns limit when you build the specification — see Specifications.

What goal-based evaluation is testing​

Goal-based evaluation tests task completion over a conversation, not the correctness of a single answer.

It helps you answer questions such as:

  • Can the agent carry a task all the way to completion, not just start it well?
  • Does it ask the right follow-up questions when information is missing?
  • Does it keep track of context and earlier turns as the conversation grows?
  • Does it converge on the goal, or does it stall, loop, or drift off task?

How the simulated user works​

During a run, the user simulator plays the human side of the conversation. It opens with the goal in mind and keeps responding to the agent across turns, steering toward the target until the goal is reached or the max-turns limit is hit. Because the agent is scored on the outcome of the whole exchange, a single good or bad turn does not decide the result — what matters is whether the conversation as a whole gets there.

Goal-based evaluation therefore needs an agent that can hold a conversation rather than answer one call at a time. The registry includes multi-turn integrations (for example OpenRouter Multi-Turn Chat Completion and the OpenDialog multi-turn agent) built for exactly this.

Reading the results​

Results use the goal-based vocabulary: each case is summarised by its Goal and Target, resolves to Pass or Fail depending on whether the goal was reached, and carries a Conversation trace you can expand to read the exchange turn by turn — with user and assistant turns labelled, alongside any judge explanation for the score. See Runs and results for how the results views work.

Why this matters​

Goal-based evaluation helps you answer practical questions such as:

  • Is this agent ready to handle real, multi-step user journeys?
  • Which kinds of tasks does it complete reliably, and which does it abandon?
  • Did a new model or prompt change make the agent worse at finishing tasks?
  • Is the agent only good at single answers, or can it actually drive a task home?

This is especially valuable when an agent's real job is a conversation — support resolution, booking, onboarding, or any flow where success only exists at the end.