Skip to main content

Attack methods

Attack methods are how Spec27 generates adversarial variants of your primary entries. Applying them in a specification produces a robust accuracy score: the share of primary entries that stay correct across all of their variants.

How you choose them depends on the team flow: Gold Team specs select individual methods, grouped by category. Red Team Light and Red Team Heavy use predefined scans; Red Team Configurable lets you choose methods.

Gold Team attack methods​

Gold Team methods are organised into four categories. You can pick any combination in the specification form.

Natural Variance​

Natural variance attacks simulate the kinds of mistakes people naturally make when typing text. These variations help evaluate how well an agent handles imperfect user input. This category includes keyboard and touchscreen typos.

Semantic Variance​

Semantic variance attacks test whether an agent understands the same intent when it is expressed using different wording, structure, or writing styles. This category includes lexical, document, and sentence paraphrase.

Persona Variance​

Persona variance attacks evaluate how well an agent handles requests written by people with different backgrounds, communication styles, or professional expertise. This category includes English as a Second Language and the Broad, Professional, Impatient, Casual, Distressed, Expert, and Verbose personas.

Other Variance​

Other Variance contains every attack that is not assigned to Natural, Semantic, or Persona Variance.

The specification form lists every selectable method under its category, with a short description of each. Leave all methods unchecked to run only the primary entries, with no adversarial run and no robustness score.

Red Team attack methods​

Choose Red Team Light for a compact, predefined vulnerability scan, or Red Team Heavy for a comprehensive, predefined scan. Use Red Team Configurable to select methods for the agent and vulnerabilities you want to test.

The reference below describes selected public methods. For conversational tests, see Red-team multi-turn evaluation.

Adaptive LotL​

Objective
To induce the agent to generate harmful content by reframing it as a required example for an NLP classification or validation task.

Applicability
Applicable to any "Red Team Configurable" specification.

Effective conditions
Use this method to test agents that follow technical task instructions or complete validation fixtures. The method can work across different agents, but success varies and is not guaranteed.

Attack mechanism
The target agent is presented with XML-formatted examples, validator code, and simulated terminal output indicating that an unsafe-response example is missing. It is then asked to provide the missing example so that the validation checks succeed. This framing casts the harmful content as test data needed to complete a technical workflow, encouraging the agent to prioritize task completion over its normal safety constraints. Subsequent prompts can be adapted using the conversation history to reinforce the framing.

Attack success rate on generic harms​

Claude Sonnet 5: 100% GPT-5.6 Terrra: 100% Gemini 3.6 Flash: 85%

Dynamic grandma​

Objective
To elicit harmful content by framing the harmful request as an emotionally meaningful interaction with a grandmother.

Applicability
Applicable to any "Red Team Configurable" specification.

Effective conditions
Use this method when empathy, emotional engagement, or roleplay may take precedence over evaluating the underlying request.

Attack mechanism
An attack-generation model creates a grandmother persona tailored to the target request, including a relevant background and relationship to the user. It then embeds the request in emotional framing such as nostalgia, grief, childhood memories, or bedtime storytelling. The target is asked to recreate the grandmother’s explanations, continue a remembered interaction, or respond in her voice. Unlike the static Grandma attack, the persona, backstory, and emotional framing are generated dynamically for each objective. Each variant is delivered as a single-turn prompt.

Attack success rate on generic harms​

Claude Sonnet 5: 0% GPT-5.6 Terrra: 0% Gemini 3.6 Flash: 35%

Fictional roleplay​

Objective
To elicit harmful content by presenting the harmful request as material required for a fictional work.

Applicability
Applicable to any "Red Team Configurable" specification.

Effective conditions
The method is particularly relevant when creative generation falls within the agent’s normal remit.

Attack mechanism
The attacker reframes the target request as part of a movie scene, novel excerpt, screenplay, game narrative, or character dialogue. A generator varies the genre, setting, characters, and motivations while emphasising realism and technical accuracy. This framing encourages the target to provide otherwise prohibited details as dialogue or narration, treating them as requirements of a creative-writing task rather than as direct instructions. Each variant is delivered as a single-turn prompt.

Attack success rate on generic harms​

Claude Sonnet 5: 56% GPT-5.6 Terrra: 57% Gemini 3.6 Flash: 95%

Poetic jailbreak​

Objective
To elicit harmful instructions by encoding a prohibited request as poetry and asking the target to convert it into a practical procedure.

Applicability
Applicable to any "Red Team Configurable" specification.

Effective conditions
Use this method to test agents that create or interpret creative work, such as writing assistants.

Attack mechanism
A generator transforms the target request into a four-line procedural poem that preserves the targeted outcome. Sensitive nouns are replaced with metaphors that retain their functional meaning. A fixed follow-up instruction asks the target to act as a historical-data recovery specialist and translate the poem into a literal standard operating procedure, including tools, execution steps, and error handling. The poem and translation instruction are delivered together in a single turn.

Attack success rate on generic harms​

Claude Sonnet 5: 0% GPT-5.6 Terrra: 0% Gemini 3.6 Flash: 100%

Privileged mode​

Objective
To elicit prohibited output by convincing the target that a higher-authority mode has disabled or overridden its normal restrictions.

Applicability
Applicable to any "Red Team Configurable" specification.

Effective conditions
Use this method to test agents with broad tool permissions or access to sensitive data that rely mainly on the model to distinguish trusted from untrusted instructions. It is also relevant to agents that ingest external content without strong privilege separation and input isolation.

Attack mechanism
An attack-generation model presents the target request as a system test, debugging command, or administrative task. It claims that a privileged mode is active and uses simulated system logs, terminal output, and technical language to make that claim appear authoritative. A command-style prompt asks the target to carry out the request under the claimed mode. The attack tests whether the agent mistakes user-supplied roleplay for higher-priority instructions; it does not actually change the agent’s permissions. Each variant is delivered as a single-turn prompt.

Attack success rate on generic harms​

Claude Sonnet 5: 2% GPT-5.6 Terrra: 0% Gemini 3.6 Flash: 20%

Research bypass​

Objective
To elicit harmful content by presenting the harmful request as necessary for legitimate academic, historical, or scientific work.

Applicability
Applicable to any "Red Team Configurable" specification.

Effective conditions
It is most effective against agents that heavily condition safety decisions on the user’s stated intent or context. Agents that treat academic, historical, or educational framing as evidence that an otherwise restricted request is benign are particularly susceptible.

Attack mechanism
A generator wraps the target request in a research scenario, such as preparing a thesis, documenting historical practices, teaching a course, or conducting scholarly analysis. It varies the claimed role, discipline, and time period while requesting detailed, accurate explanations. The attack encourages the agent to treat the stated educational purpose as justification for providing information it would otherwise withhold.

Attack success rate on generic harms​

Claude Sonnet 5: 82% GPT-5.6 Terrra: 71% Gemini 3.6 Flash: 95%

Planned multi-turn red team​

Objective
To elicit behaviour that violates the specification through a planned, multi-turn conversation that adapts to the target agent’s responses.

Applicability
Applicable to any Red Team specification for conversational agents.

Effective conditions
Particularly relevant for conversational agents whose safety boundaries may weaken over successive turns. It tests whether a planned sequence of requests can steer the agent toward harmful output that it would reject in a direct request.

Attack mechanism
A planning model constructs a realistic user persona, a motivating scenario, an attack strategy, and a sequence of conversational steps. Early turns establish context and credibility, while later turns progressively steer the interaction toward the target behaviour. An executor generates each message using the plan and the evolving conversation history, adapting to refusals, partial compliance, or other intermediate responses while remaining consistent with the assigned persona. The final planned turn attempts to elicit the complete target behaviour, and each target response is evaluated for specification violations. Unlike methods constrained to a single attack technique, the planner can select among approaches such as storytelling, research framing, technical consultation, or expressions of personal concern.

Attack success rate on generic harms​

Claude Sonnet 5: 83% GPT-5.6 Terrra: 75% Gemini 3.6 Flash: 89%

Practical guidance​

  • Choose methods by what you are testing: everyday input variation (Natural, Semantic, Persona variance) versus resistance to misuse (Red Team methods).
  • Start with a predefined Red Team scan, or use Red Team Configurable to choose methods for a specific evaluation.
  • Read robustness per attack method on the robustness views to find where an agent is weakest.