Skip to main content

Mental Model

Spec27 is easiest to understand as a set of reusable assets that come together in a specification. You connect an agent, prepare a specification, run it from an eval, and review the results later. Each run snapshots the specification as an immutable version, so those results stay tied to the exact setup that produced them.

Organization​

The organization is the top-level shared workspace. Memberships, roles, invite links, and usage limits live here.

Project​

The project is the main container for one evaluation stream. It holds the assets and results that belong together.

Assets inside a project​

  • Agents hold the runnable logic you want to test. You connect them with Agent Builder, a registry integration, or by writing the code.
  • Secrets provide protected runtime values when an agent needs them.
  • Specifications are the central authoring unit. A specification brings together its test entries, the evaluation method, judging configuration, and the adversarial coverage from attack methods.
  • Evals pair agents with specifications into a saved, runnable setup.

You author entries and scoring directly inside the specification. Testsets and judges are internal building blocks that support the specification, but you do not manage them separately.

The specification is the unit of meaning​

A specification separates what you are preparing from what has been run:

  • Editing a specification updates a working draft.
  • Each eval run snapshots the draft as an immutable version, reusing the newest version when nothing changed.
  • A past run keeps the version it used, so editing the draft later never changes its meaning.

This is what lets you ask "what exactly was run?" after the fact and get a stable answer. See Specifications for the full flow.

Runs and results​

  • A run is a single execution of an eval (or a quick agent preview).
  • Results are the recorded outputs, scores, statuses, and logs from that run, summarised as clean and robust accuracy.

Gold Team and Red Team use the same workflow​

Both move through the same chain of assets, specifications, evals, runs, and results.

The workflow at a glance​

Organization → Project → Agent + Specification (entries · method · judge · attacks) → Eval → Run → Results