End-to-end flow
This page narrates the full journey through Spec27, from creating a project to reading results. Each step links to the guide that covers it in depth.
The workflow at a glance:
Organization → Project → Specification → Eval → Run → Results
1. Project
You work inside a project, the container for one evaluation stream. A project holds your agents, specifications, evals, and the results they produce. Everything below lives in a single project.
2. Specification
The specification is where you author what to test and how to score it. You build it in the specification editor, which is organised into a few tabs:
- Entries: the test cases. Add them manually or generate them. You can import a CSV when the schema uses one scalar string. Adversarial variants are produced from your primary entries using attack methods and grouped under the entry they derive from.
- Schema: the contract for scalar or structured inputs.
- Knowledge: context for the agent and context for judging.
- Evaluation: the scoring method, including judge-based scoring when correctness needs interpretation.
A specification has one of three families, chosen at creation and then locked:
- Gold Team for single-turn desirable behaviour,
- Goal Gold Team for multi-turn goal-based tasks,
- Red Team for adversarial, failure-seeking evaluation.
Your edits are saved as a working draft as you go. There is no separate save step: when a run starts, Spec27 snapshots the draft as an immutable version of the entries, schema, context, and evaluation setup.
3. Eval
An eval pairs one or more specifications with the agents you want to test. Every run of the eval tests the specifications as they are when the run starts.
4. Run
Running an eval snapshots each specification's draft as a version and runs it against every selected agent. A run reuses the newest version when the draft has not changed since. A run fans out into per-specification work, sending each entry to the agent (or, for goal-based and multi-turn specs, driving a simulated conversation) and scoring the output with the specification's evaluation method.
5. Results
Results record the outputs, scores, statuses, and logs from a run. They are summarised as clean and robust accuracy: clean accuracy over your primary entries and robust accuracy over the adversarial variants, with a per-entry breakdown you can drill into. Each run shows the version it used, so you can always answer "what exactly was run?" from the results.