Skip to main content

End-to-end flow

This page narrates the full journey through Spec27, from creating a project to reading results. Each step links to the guide that covers it in depth.

The workflow at a glance:

Organization → Project → Specification → Eval → Run → Results

1. Project​

You work inside a project, the container for one evaluation stream. A project holds your agents, specifications, evals, and the results they produce. Everything below lives in a single project.

2. Specification​

The specification is where you author what to test and how to score it. You build it in the specification editor, which is organised into a few tabs:

  • Entries: the test cases. Add them manually or generate them. You can import a CSV when the schema uses one scalar string. Adversarial variants are produced from your primary entries using attack methods and grouped under the entry they derive from.
  • Schema: the contract for scalar or structured inputs.
  • Knowledge: context for the agent and context for judging.
  • Evaluation: the scoring method, including judge-based scoring when correctness needs interpretation.

A specification has one of three families, chosen at creation and then locked:

Your edits are saved as a working draft as you go. There is no separate save step: when a run starts, Spec27 snapshots the draft as an immutable version of the entries, schema, context, and evaluation setup.

3. Eval​

An eval pairs one or more specifications with the agents you want to test. Every run of the eval tests the specifications as they are when the run starts.

4. Run​

Running an eval snapshots each specification's draft as a version and runs it against every selected agent. A run reuses the newest version when the draft has not changed since. A run fans out into per-specification work, sending each entry to the agent (or, for goal-based and multi-turn specs, driving a simulated conversation) and scoring the output with the specification's evaluation method.

5. Results​

Results record the outputs, scores, statuses, and logs from a run. They are summarised as clean and robust accuracy: clean accuracy over your primary entries and robust accuracy over the adversarial variants, with a per-entry breakdown you can drill into. Each run shows the version it used, so you can always answer "what exactly was run?" from the results.