Skip to main content

Generating testset entries

When you need more coverage than you can write by hand, Spec27 can generate synthetic entries for you, based on the entries the specification already has and the instructions you provide. Generation is available for Gold Team, Goal Gold Team, and Red Team Configurable specifications.

Generated entries are added as primary entries of the specification alongside entries you wrote manually. Scalar string schemas also support imported entries. See Specifications for an overview of a specification's structure.

Generate entries​

  1. Open the specification.

  2. In the entries toolbar, choose Add entries → Generate entries.

  3. Set Number of queries to generate between 1 and 100. The default is 10.

  4. For a Gold Team or Goal Gold Team specification, optionally steer the output:

    • Which aspect of the agent would you like to test in this testset?
    • Any custom instructions

    These answers apply only to this generation run.

  5. Review the proposed coverage taxonomy. You can add, remove, rename, and redistribute its generation targets.

  6. Start the generation. It runs in the background. The editor shows progress while it works, and the new entries appear when it finishes.

For a Red Team Configurable specification, step 4 asks you to select vulnerability classes instead. Spec27 preselects classes from Task context. If the specification has no Task context, it selects General Harms. Task-specific Harms, Personally Identifiable Information, and Context Drift require Task context, so you cannot select them until you add it. General Harms and System & Security Exposure remain available without Task context. You can also describe one custom class instead of, or alongside, the predefined classes. Spec27 names the custom class and proposes its subcategories for review.

Task-specific entries use facts from Task context. Personally Identifiable Information entries use only the personal data types stated there. Context Drift entries test whether the agent loses, distorts, or deprioritizes instructions and constraints stated in Task context over an extended interaction. General Harms and System & Security Exposure entries do not depend on Task context.

Task-specific Harms does not cover disclosure or inference of personally identifiable information. Select Personally Identifiable Information when you want the testset to cover those outcomes. Task-specific Harms can still cover other harmful outcomes in workflows that process personal data.

The review step explains when a selected or custom class lacks enough context and omits it from generation. Your custom class description applies only to the current generation flow.

Each generated Red Team expected output identifies the harmful response that the agent must not provide. It does not require the agent to refuse explicitly, apologize, explain, redirect the user, or offer alternative help.

Review generated entries​

Generation is a starting point, not a finished testset. After the run completes, review the new entries the same way you would review any entry in your specification:

  • Check that inputs are representative and expected outputs are correct.
  • Edit or remove entries that do not fit.
  • Keep the set focused. A smaller, well-curated testset is easier to reason about than a large, noisy one.