AI Red Teaming: Methods, Tools & Examples (2026)

AI red teaming is structured adversarial testing that probes a deployed AI system for security and safety weaknesses before an attacker or an ordinary user finds them. Operators emulate a realistic actor, attack the model, the application, and the agent, and document what the system did.

Most AI red teaming programs running today were scoped against a risk list that OWASP replaced in August 2026. Excessive Agency climbed from sixth to third in that revision, which tells you where the damage moved.

TL;DR: AI red teaming

• Single-turn probing understates your exposure badly. The X-Teaming research took attack success from 19.5% at two turns to 92.7% at eight turns, easing to 87.8% at ten as long conversations dilute context, against 12.5% for the GCG single-turn baseline on GPT-4o under HarmBench.
• Red teaming and safety benchmarking answer different questions. Benchmarks compare models on a fixed dataset, while red teaming finds harms nobody has written a dataset for yet.
• Adversarial testing carries legal weight in the EU. Article 55 obliges providers of general-purpose models with systemic risk to conduct and document it.

What is AI red teaming?

AI red teaming is structured adversarial testing that probes an AI system for security and safety weaknesses before an attacker or an ordinary user encounters them. Operators emulate a realistic actor, run attacks against the deployed system, and document what the system did.

In practice, it is the same discipline security teams have run for decades, pointed at a system whose outputs are probabilistic.

Microsoft's red team emulates both hostile actors and benign users who trip a harm by accident.

Most safety research assumes hostile intent. A customer who stumbles into a harmful response was never attacking you, and your logs will not flag them.

A red team engagement targets the model, the application wrapped around it, or the agent acting through it. Those three produce different findings against the same underlying model.

How red teaming differs from safety benchmarking

Benchmarks measure preexisting notions of harm. That is their strength for comparison and their limitation for discovery.

When a model gains a capability nobody has catalogued, no dataset exists to score it against. Microsoft's group treats those gaps as the assignment, helping define novel harm categories and building probes to measure them.

A benchmark tells you how your model ranks, and red teaming tells you what your deployment does that nobody thought to measure.

Both belong in a mature program. Findings from adversarial work become tomorrow's benchmarks.

The three layers of an AI red team engagement

Model layer

The model layer covers jailbreaks, refusal bypass, encoding attacks, and dangerous capability probing. Scope it to what the model can actually do.

A capability-constrained model does not need testing for attacks it cannot parse. Microsoft found that larger models understand base64 and ASCII art encodings that smaller ones simply do not process, so encoding attacks against a small model waste your operators' time.

Microsoft also found that larger Phi-3 models adhered better to user instructions. The capability that makes them useful is the one that makes crafted malicious instructions land.

Application layer

The application layer is where most exploitable findings live, because the model is one component inside a system with databases, filters, retrieval, and cloud resources around it.

One Microsoft operation chained three techniques together. Operators ran low-resource language prompt injections to identify internal Python functions, used a cross-prompt injection to generate a script calling them, then executed it to exfiltrate private user data.

Every prompt in that chain was hand-written.

Some of the highest-severity findings in AI systems have nothing to do with the model. Microsoft lists outdated dependencies alongside improper error handling, missing input and output sanitization, and credentials in source as the engineering practices behind these findings.

Agent layer

Agents hold credentials, call tools, persist memory, and act across many steps. OWASP built a separate taxonomy for them, running ASI01 through ASI10, because the LLM list covers those risks incompletely.

A compromised chatbot returns bad text, but a compromised agent sends the email, moves the money, or writes to the database.

An attacker who seeds an agent's persistent store influences decisions in sessions that have not happened yet, so your test has to span sessions to catch it.

Voice agents add a layer, since audio conditions change what the model receives. Our guide to red-teaming chat and voice AI agents walks through the bias, toxicity, and jailbreak scenario families that surface there.

The 2026 risk taxonomies that define red team scope

OWASP Top 10 for LLM Applications 2026

The current list runs in this order:

• LLM01 Prompt Injection
• LLM02 Sensitive Information Disclosure
• LLM03 Excessive Agency
• LLM04 Supply Chain
• LLM05 Data and Model Poisoning
• LLM06 Unbounded Consumption
• LLM07 Misinformation
• LLM08 Hidden Context Exposure
• LLM09 Vector and Embedding Weaknesses
• LLM10 Improper Output Handling

Eight of the ten entries moved position. Excessive Agency was the most consequential change, climbing from sixth to third, driven by agentic deployments doing real damage in production. Improper Output Handling fell hardest, from fifth to tenth.

LLM08 was renamed and widened from System Prompt Leakage. Hidden context now covers developer instructions, tool schemas, and policy logic, so your extraction tests should target all of it.

OWASP Top 10 for Agentic Applications

Announced in December 2025, the agentic list covers goal hijacking, tool use, identity, supply chain, code execution, memory, inter-agent communication, cascading effects, human-agent trust, and rogue agents.

It extends the LLM list and replaces nothing. Most agent systems are also LLM applications and inherit both sets of risk.

OWASP also published a dedicated red teaming taxonomy, Solutions Landscape: Red Teaming Taxonomy, on 28 June 2026, which is the closest thing the field has to shared vocabulary for classifying what your operators actually did.

MITRE ATLAS and threat model ontologies

A shared schema makes findings trackable across engagements.

Microsoft models every attack with five components: system, actor, tactics and techniques, weakness, and impact. Tactics and techniques map to MITRE ATT&CK and the ATLAS matrix wherever possible.

Start from downstream impact: List what could go wrong for a real person, then trace which failures make it possible.

The same model carries different risk as a writing assistant than as a summarizer of patient records. Beginning with attack strategies produces clever findings with no connection to anything a customer would suffer.

Seven AI red teaming methods

Pick by what your system exposes.

1. Single-turn prompt attacks

What it does: Sends one crafted input designed to defeat a guardrail immediately. Includes instruction override, role reassignment, delimiter confusion, and persona hacking.

Best for: Regression gates in CI, where speed matters more than depth.

Treat single-turn results as a floor on your exposure. A model that refuses every one-shot attempt has told you very little.

2. Multi-turn escalation

What it does: Distributes intent across a conversation, establishing rapport before the request that matters.

Best for: Any agent with a conversational interface.

X-Teaming reported up to 98.1% attack success, peaking against DeepSeek V3, with 96.2% against Claude 3.7 Sonnet, a model regarded as resistant to single-turn attacks. Results varied by model; Claude 3.5 Sonnet held at 67.9%.

Read those numbers with the method attached. SEMA (Feng et al., SEMA, ICLR 2026, arXiv:2602.06854) reran X-Teaming's successful attempts through the same judge and found 15.1% no longer succeeded. It also noted the headline rate reflects ten sampled strategies per input.

Multi-turn still dominates single-turn by a wide margin. The precise figure depends on how many attempts you allow. A turn-by-turn breakdown of the same effect is published here.

3. Indirect prompt injection

What it does: Plants instructions in content the system ingests. Documents, web pages, tickets, calendar invites, tool responses, and retrieved records all reach the same token stream.

Best for: RAG pipelines and any agent that reads untrusted input.

Direct injection assumes the attacker talks to your agent. Indirect injection assumes they only have to write something your agent will eventually read.

What it looks like in practice: A support ticket contains instructions addressed to the agent that will summarize it. A vendor PDF in your RAG index carries a line telling the retriever to ignore prior instructions.

How to test it: Seed your own corpus with tagged payloads and check whether they survive retrieval into the model's context.

4. Multimodal and encoding attacks

What it does: Moves the payload into a channel with weaker safety training. Images, audio, and unusual text encodings all qualify.

Best for: Vision and audio models, and any system accepting file uploads.

Multimodal inputs bypass text-channel guardrails. A model that refuses a request in text will often comply when the same instruction arrives overlaid on an image.

5. Automated attacker-model loops

What it does: Puts one model in the attacker seat against your system, scoring each result and regenerating with feedback.

The 2026 version of this is agent-orchestrated: the attacker is handed an objective in natural language and selects its own techniques, composes transforms, and reports structured findings. PyRIT's orchestrators and Promptfoo's Hydra strategy both work this way.

Best for: Covering more of the risk surface than manual probing reaches.

Cost: An automated loop is cheap per attempt and expensive per false positive, because every flagged result still needs a human verdict. Budget operator hours against attempt volume as opposed to tool licences.

Automation handles volume and the non-deterministic behavior of the target. Running an attack once tells you it happened, and running it fifty times tells you how likely it is.

6. Tool and permission abuse

What it does: Enumerates the agent's tools and credentials, then tests authority scope, confirmation bypass, and tool chaining.

Best for: Agents with write access to anything.

Test what the agent can do when convinced it should. Your documentation describes intent, and the agent describes behavior.

7. Human expert probing

What it does: Applies judgment that automation cannot supply. Domain expertise, cultural context, and emotional intelligence.

Best for: Specialized domains and subjective harms.

Model judges work well for detecting hate speech or explicit content. They are less reliable in medicine, cybersecurity, and CBRN, where only subject matter experts can assess the output accurately.

Microsoft also tests how systems respond to users in distress, working with specialists in psychology, sociology, and medicine on guidelines for psychosocial harms.

That work exposes operators to disturbing content repeatedly. The group maintains processes to let people disengage and access mental health support.

AI red teaming examples from documented operations

Microsoft published five case studies from its own operations, each mapped to the same five-part threat model: system, actor, tactics and techniques, weakness, and impact.

Image overlay defeating a vision model's guardrails

Operators overlaid malicious instructions onto an image and sent it to a vision language model API. Text-channel refusals held, but the image channel complied, revealing insufficient safety training on visual input.

An automated scam pipeline built from off-the-shelf parts

Investigators jailbroke a model into supplying persuasion tactics, then connected text-to-speech and speech-to-text so a target could converse naturally. The output was a working proof of concept for automated voice scams.

Gender bias measured through repetition

Operators wrote prompts describing people without specifying gender, sent each one fifty times to a text-to-image generator, and manually labeled the results. A single generation proves nothing about a probabilistic system.

Server-side request forgery in a video service

An outdated FFmpeg component let an attacker craft malicious video files, reach internal resources, and escalate privileges. The remediation was a version upgrade and component isolation.

Chatbot responses to users in distress

Operators ran multi-turn conversations as users expressing grief, seeking mental health advice, or signaling intent for self-harm. No adversarial intent existed anywhere in the scenario, and the potential harm was still severe.

AI red teaming tools in 2026

Open-source frameworks

PyRIT is Microsoft's Python framework, released under MIT. It ships prompt datasets, converters for encodings, automated attack strategies including TAP, PAIR, and Crescendo, and scorers that handle multimodal output. Microsoft's own operators use it heavily.

garak is NVIDIA's vulnerability scanner, Apache-2.0 licensed and actively maintained. It runs probe modules against a target model and reports which ones landed.

Meta's PurpleLlama collects tools for assessing and improving LLM security, including the Llama Guard classifiers.

Commercial platforms

Promptfoo, Giskard, DeepEval, Braintrust, and LangWatch cover open-core evaluation and security scanning with paid tiers. Cyara and Maxim AI serve enterprise contact centers and multimodal agent evaluation, respectively. F5 and Palo Alto Networks sell red teaming inside broader security suites.

Several of these now ship multi-turn strategies of their own; Promptfoo alone documents Crescendo, GOAT, and Hydra. What separates Cekura is the surface: sustained multi-turn simulation against live voice and chat agents over the phone instead of text transcripts alone.

Automation covers ground, and it does not replace operators. Microsoft is explicit that PyRIT exists to augment human judgment, since prioritizing risks, designing system-level attacks, and defining new harm categories all still require people.

Adversarial testing requirements under the EU AI Act

Article 55(1)(a) of the EU AI Act obliges providers of general-purpose AI models with systemic risk to perform model evaluation using standardized protocols.

That includes adversarial testing to identify and mitigate systemic risks, with 55(1)(c) requiring the results to be documented and reported.

The provision started to apply on 2 August 2025. Providers may rely on codes of practice to demonstrate compliance until a harmonized standard is published, and anyone outside an approved code has to show alternative adequate means.

The documentation is the obligation, alongside the testing itself. An engagement with no artifact trail satisfies nothing. Keep threat models, run records, findings, remediation evidence, and proof that fixes were regression-tested on later releases.

The same article requires incident reporting to the AI Office and adequate cybersecurity protection for the model and its physical infrastructure.

Platforms like Cekura produce that trail as a by-product of running the suite, since every scenario, score, and rerun is recorded against a build.

Frameworks outside the EU

The US has no statutory red teaming duty. NIST's AI Risk Management Framework and its Generative AI Profile treat adversarial testing as a measure-function practice, and ISO/IEC 42001 expects it as evidence inside an AI management system.

Both are voluntary, and both are what an enterprise procurement questionnaire will ask you about.

Building an AI red teaming program: eight decisions

These are program-level decisions. For the hands-on sequence against a live voice agent, follow our 8-step penetration test walkthrough.

1. Start from downstream impact: List what could go wrong for a real person, then work backward to the paths an adversary would take. The same model carries different risk as a writing assistant than as a summarizer of patient records.

2. Scope to system capability: Skip attack classes the system cannot process. Spend the recovered hours on attacks it can.

3. Prioritize simple techniques: Real attackers prompt engineer far more often than they compute gradients. Gradient methods are expensive and usually need model access that commercial systems do not grant.

4. Attack the whole system: Strategies aimed only at the model miss weaknesses in filters, databases, and cloud resources. Strategies ignoring the model miss the rest.

5. Automate for coverage: Use a framework to generate variations, orchestrate attacks, and score responses at volume. Keep operators on judgment work.

6. Bring in domain experts: Specialized content needs specialized assessment. Note honestly where your scoring is unreliable.

A working team is four roles: a security engineer for the application surface, an ML practitioner for model behavior, a domain expert for the content, and an operator lead who owns the ontology and the report.

7. Run repair cycles: Microsoft safety-aligned the Phi-3 models through repeated rounds of red teaming and mitigation. Purple teaming, applying offensive and defensive work together, tends to raise attack cost further than a single round.

8. Regression-test every mitigation: Fixes introduce new weaknesses. A finding is closed once the attack that produced it runs clean on a later build. Our write-up on automated QA, red teaming, and regression for Retell agents shows that loop running against a live agent.

Scoring and reporting red team results

Attack success rate is the standard headline metric, expressed as the percentage of attempts achieving their target behavior under a defined scoring rubric.

Report the attempt budget alongside it. ASR at one attempt and ASR at ten attempts describe different threat models, and quoting one as the other overstates or understates your position.

For conversational agents, a graded severity scale carries more signal than a binary verdict. Cekura scores each attack from 1 to 5, where 1 is complete refusal, 3 is partial engagement, and 5 is full compliance.

Scores of 1 or 2 count as successful defense. Scores of 4 or 5 mark weaknesses needing work.

A graded scale separates a clean refusal from a near miss. Binary scoring hides the agent that almost gave way, which is the one that will give way after your next prompt edit.

AI red teaming best practices

Test across turns as well as inputs: Conversation-spanning attacks outperform single-shot attempts by margins large enough to change your risk picture entirely.

Probe for accidental harm: Ordinary users trigger harmful output without trying, and that scenario is usually worse for you than a deliberate attack.

Assume untrusted input yields arbitrary output. Given fundamental limits on alignment, treat any system fed untrusted data as capable of emitting anything it holds.

Version your scenarios as code: Adversarial suites belong in CI next to your unit tests, running on every prompt and model change.

Track findings against a shared schema: Mapping to ATLAS or an internal ontology makes results comparable across engagements and legible to auditors.

Support your operators: Sustained exposure to disturbing content is an occupational hazard with a duty of care attached.

Red team the program as well as the model: Majumdar, Pendleton and Gupta (Red Teaming AI Red Teaming, CAMLIS 2025, arXiv:2507.05538) argue the field has narrowed to hunting model-level flaws and misses the sociotechnical failures that emerge between models, users and environments. Their fix is two levels of engagement, one macro and one micro.

How Cekura automates red teaming for conversational AI agents

Cekura generates sustained adversarial conversations spanning 5 to 10 turns. Each one adopts a believable persona, a QA engineer or a compliance auditor, that builds trust before attempting exploitation.

Attacks run across six categories covering system prompt leaks, data leaks, harmful content, biased output, unauthorized actions, and off-task drift.

Text mode runs up to three optimization cycles, regenerating with feedback when an attempt scores below 4. Voice mode generates once, tuned for first-attempt success given call cost and latency. The scenario library behind it holds 10,000+ scenarios.

Relevant capabilities across the lifecycle:

• Pre-production: Automated evaluator generation builds scenarios from your agent's own context and knowledge base.
• Infrastructure: Interruption, background noise, latency, and endpointing checks under real call conditions.
• Observability: Production call scoring against the same criteria as your pre-launch suite, with alerts on quality drift.

Native integrations work out of the box for Retell, Vapi, ElevenLabs, LiveKit, Pipecat, Bland, and more. You add a testing and monitoring layer over the stack you already run.

Cekura supports SOC 2, HIPAA, and GDPR compliance, covering transcript redaction, role-based access, and audit trails. Healthcare operators run adversarial suites under HIPAA, where an agent talked into confirming an appointment it never booked costs a patient a visit.

Where to start with AI red teaming

Pick the layer matching your exposure:

• Start at the model layer if you fine-tune or host your own weights
• Start at the application layer if you wrapped a hosted model in retrieval and filters
• Start at the agent layer if anything you shipped holds credentials and takes actions

Then make it repeat. A single engagement produces a snapshot of a system that changes with every prompt edit, and the documentation regulators want is a trail across releases. Cekura gates that trail on every deploy.

Ready to see what your agent does under sustained adversarial pressure? Book a demo and watch Cekura run multi-turn red teaming, persona simulation, and production monitoring against your own agent.

Frequently asked questions

What is AI red teaming?

AI red teaming is structured adversarial testing that finds security and safety weaknesses in an AI system before real users hit them. Operators emulate realistic actors, attack the deployed system end to end, and document what happened. It covers hostile attackers and ordinary users alike.

What is the difference between AI red teaming and penetration testing?

The main difference between AI red teaming and penetration testing is scope. Penetration testing hunts known vulnerability classes across infrastructure. Adversarial AI testing probes behavior through open-ended interaction, covering model weaknesses, harmful content, and agent misuse.

Our 8-step penetration test walkthrough for voice AI agents covers the hands-on version.

Is red teaming required by law?

Yes, in specific cases. The EU AI Act requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing under Article 55, which started to apply on 2 August 2025.

Elsewhere, it remains a recommended practice under voluntary frameworks, with no statutory duty attached.

How often should you red team an AI system?

Run adversarial suites on every prompt change, model swap, and knowledge base update, with deeper manual engagements scheduled periodically. Continuous testing matters because mitigations introduce new weaknesses, and a system that passed last month may respond differently after one edit.

Can AI red teaming be fully automated?

No. Automation covers volume while people supply judgment.

Frameworks generate prompts, orchestrate attacks, and score responses at a scale manual work cannot reach. What they cannot do is decide which risks matter for your product or recognize a harm category nobody has named yet.

What is multi-turn red teaming?

Multi-turn red teaming spreads an attack objective across a whole conversation. The attacker adopts a persona, establishes context, and escalates gradually. Published research puts multi-turn attack success far above single-turn baselines against the same models.