Specifications
A specification is the central authoring unit in Spec27. It brings together everything needed to evaluate a behaviour: the primary entries, the evaluation method, any scoring configuration, and the adversarial coverage. You prepare a specification once and reuse it across many evals.
Choose a specification type
When you create a specification, you first pick the team flow on the Choose Specification Type page so the form stays focused:
- Gold Team: desirable behaviour, correctness, and robustness.
- Red Team: misuse, failure modes, and attack-oriented evaluation.
- Goal Gold Team: simulator-driven multi-turn goal completion.
Author the specification
After choosing a type, enter the specification name and create it. The specification editor is organised into these sections:
- Specification details: a Name and optional Additional context that evaluators should apply. The type is fixed once chosen.
- Entries: author or generate test case entries in the Entries tab. CSV import is available when the schema uses one scalar string, but not for structured entries. Red Team specs include a PII attestation for the entries. Use testset generation to help populate your entries.
- Schema: use the visual editor or raw JSON editor to define the input. The schema can use one scalar string or a structured object with typed fields. Gold Team schemas can mark text fields as perturbable. Goal Gold Team and Red Team schemas identify one text field as the Goal. If a schema change leaves entries Off-schema, repair those entries before you preview or run the specification.
- Evaluation settings: choose how outputs are scored:
- Evaluation method: Strict equality, Permitted values, or Judge. For Permitted values you list the accepted values, one per line. Red Team is always Judge scored.
- Execution mode: Batch or Iterative, when applicable.
- Judge: the judge configuration to use when the method is Judge. See evaluation methods for scoring options.
- For goal-based and red-team multi-turn specs, a User Simulator and Max turns that bound the conversation. Some Red Team attack methods, such as Adaptive LotL, generate follow-up turns without the selected user simulator.
- Attack coverage: define adversarial variants:
- Gold Team specs select individual attack methods, grouped by category. Leave them unchecked to run only the primary entries, with no robustness score.
- Red Team specs pick an attack suite, Light Suite or Heavy Suite, instead of individual methods.
The editor persists these changes in the working draft.
Editing, versions, and runs
A specification separates what you are preparing from what has been run:
- Editing changes the draft. Updating the name, entries, evaluation settings, or attack coverage changes the working draft. There is no save step.
- Running takes a snapshot. When an eval run starts, Spec27 snapshots each specification's draft as an immutable version. If the draft has not changed since the newest version, the run reuses that version; otherwise it creates the next one, such as v2.
- Results keep their version. The specification run page and the eval's run history show the version each run used, and later edits to the draft do not change past runs. Select the version, such as v2, to open a read-only page with its entries, variants and settings exactly as that run used them.
An eval always runs your specifications as they are now. If a draft cannot run, for example because an entry is Off-schema, the run confirmation names the specification and the reason, and the run does not start.
Track preparation status
The specification detail page shows its preparation status:
- Preparing: Spec27 is preparing the specification (including any adversarial coverage).
- Ready: the specification can be used in eval runs.
- Failed: preparation hit an error; use Retry preparation to try again.
The detail page also shows the evaluation method, selected entries, selected attack methods, additional context, and a link to results.