Skip to main content

End-to-end flow

This page narrates the full journey through Spec27, from creating a project to reading results. Each step links to the guide that covers it in depth.

The workflow at a glance:

Organization → Project → Specification → Eval → Run → Results

1. Project​

You work inside a project, the container for one evaluation stream. A project holds your agents, specifications, evals, and the results they produce. Everything below lives in a single project.

2. Specification​

The specification is where you author what to test and how to score it. You build it in the specification editor, which is organised into a few tabs:

  • Entries: the test cases. Add them manually or generate them. You can import a CSV when the schema uses one scalar string. Adversarial variants are produced from your primary entries using attack methods and grouped under the entry they derive from.
  • Schema: the contract for scalar or structured inputs.
  • Knowledge: context for the agent and context for judging.
  • Evaluation: the scoring method, including judge-based scoring when correctness needs interpretation.

A specification has one of three families, chosen at creation and then locked:

Your edits are saved as a working draft as you go. When the draft is ready, you save a version, an immutable snapshot of the entries, schema, context, and evaluation setup. Versions are what evals run, so a saved version always means the same thing later.

3. Eval​

An eval pairs one or more specifications with the agents you want to test. Each eval pins to a specific saved specification version. Editing a draft afterwards does not change a pinned eval. You explicitly bump the eval to a newer version when you want it to pick up your changes. This keeps past results reproducible.

4. Run​

Running an eval executes every pinned specification against every selected agent. A run fans out into per-specification work, sending each entry to the agent (or, for goal-based and multi-turn specs, driving a simulated conversation) and scoring the output with the specification's evaluation method.

5. Results​

Results record the outputs, scores, statuses, and logs from a run. They are summarised as clean and robust accuracy: clean accuracy over your primary entries and robust accuracy over the adversarial variants, with a per-entry breakdown you can drill into. Because the run used a pinned version, you can always answer "what exactly was run?" from the results.