Evals
An eval is a saved, reusable setup that pairs one or more agents with one or more specifications. It is the named workflow your team runs again and again.
Create an eval
- Open Evals and add one.
- Under Eval details, give it a Name and an optional Description.
- Under Coverage, select the Agents and Specifications the eval should use.
- Select Create Eval.
Review and run
The eval detail page summarises the setup at a glance: Agents, Specifications, Schedule, and Results. You can confirm it matches your intent. When ready, select Run eval to start a run, then open View results to follow it. See Runs and results for reading the outcome.
Schedule recurring runs
An eval can run on a schedule. The Schedule details show whether it is Enabled or Disabled, the cadence (for example daily or weekly with chosen days), the local time and timezone, the next expected run, and the latest scheduled result. Use this for recurring regression checks without starting each run by hand.
If the eval contains a Red Team specification, your organization needs at least 750 available credits for every run. Human verification does not bypass this credit requirement.
In staging and production, enabling or saving an enabled Red Team schedule requires human verification. After you verify the schedule, its occurrences start automatically without asking you to verify each one. Spec27 still checks for at least 750 available credits when each occurrence starts. Disabling the schedule clears its verification, so re-enabling it requires a new check.
See Red Team testing for the manual run, preview, and verification flow.