Skip to main content

Glossary

Adversarial robustness​

The percentage of evaluated Red Team primary entries whose direct prompt passes and whose scored attack variations have no failures. The overall score uses the lowest defined attack-method score.

Agent​

Runnable logic that processes an input.

Attack method​

A transformation that creates adversarial variations from primary entries.

Baseline performance​

The percentage of scored Gold Team primary entries the agent passes.

Baseline robustness​

The percentage of scored Red Team primary entries whose direct prompts the agent handles correctly.

Clean accuracy​

The percentage of scored primary entries with a correct result. This same measure is Baseline robustness for Red Team and Baseline performance for Gold Team and Goal Gold Team.

Credit​

A plan unit consumed by metered product work.

Credit usage​

Credits reserved or consumed by an organization during its current plan period.

Entry​

A single test case or example within a specification that defines an input and expected behavior.

Entry category​

An optional label that groups related primary entries for analysis. Red Team shows entry categories as vulnerability classes.

Eval​

A named evaluation setup that connects agents and specifications.

Eval run​

One execution of an eval across its agents and specifications. Each specification runs as the version snapshotted when the eval run starts.

Gold Team​

Evaluation work focused on desirable behavior, correctness, and robustness.

Judge​

A scoring configuration used when correctness requires interpretation.

Multi-turn eval​

An eval in which an agent and a test exchange more than one turn.

Organization​

The top-level shared workspace for members, projects, plan limits, and credit usage.

Performance under perturbations​

The percentage of evaluated Gold Team primary entries with a correct primary result and no scored failure among their transformed variations. The overall score uses the lowest defined perturbation-method score.

Perturbation method​

A way of transforming a Gold Team primary entry to check whether the agent still produces the expected result.

Plan​

The resource limits and credit allowance assigned to an organization.

Plan limit​

The maximum resource count or credit usage allowed by a plan.

Playground​

An exploratory surface for trying configurations and generating derivative entries.

Primary entry​

The original test case in a specification, before an attack or perturbation method creates variations.

Project​

The main container for assets and results.

Project API key​

A secret credential that grants API access to all or selected evals in one project.

Red Team​

Evaluation work focused on misuse, harmfulness, jailbreaks, and failure-seeking behavior.

Resource limit​

The maximum count for one plan-governed organization resource.

Result​

A recorded output, score, or status produced during a run.

Robust accuracy​

The percentage of evaluated primary entries with a correct primary result and no scored failure among their variations. The lowest defined method score is Adversarial robustness for Red Team or Performance under perturbations for Gold Team and Goal Gold Team.

Run​

One product execution that produces a result.

Scored result​

A result that completed execution and judging without an error.

Secret​

A protected project value used by an agent at runtime.

Specification​

A reusable evaluation recipe containing entries, methods for creating variations, and scoring configuration.

Specification run​

One execution of a specification version against one agent within an eval run.

Specification version​

An immutable snapshot of a specification's draft, taken when a run starts. A run reuses the newest version when the draft has not changed since.

Vulnerability class​

The Red Team name for an entry category covering related risks or harmful behaviors. It describes what is being tested, not a confirmed vulnerability.