Get Early Access
HomeLearnAI Red Teaming

AI RED TEAMING

What is AI red teaming?

AI red teaming is the practice of testing AI systems with adversarial scenarios to uncover unsafe behavior, security weaknesses, and control failures before attackers or real-world conditions expose them.

For AI agents, red teaming should test not only model responses but also identities, permissions, tools, MCP connections, data access, and actions.

Explore AI Red TeamingExplore AI Agent Security
TEST THE FULL PATHFrom adversarial input to observable action.
INPUTMODELIDENTITYTOOLACTION

THE DEFINITION

AI red teaming tests what an AI system can do under pressure.

AI red teaming is the practice of deliberately testing AI systems with adversarial inputs, attack scenarios, and misuse cases to identify security weaknesses, unsafe behavior, and control failures.

AI red teaming can evaluate models, applications, agents, tools, permissions, data access, and runtime actions.

  • Prompt injection
  • Indirect prompt injection
  • Jailbreak resistance
  • Sensitive data exposure
  • Excessive permissions
  • Excessive agency
  • Tool misuse
  • MCP risks
  • Unsafe multi-step workflows
  • Approval bypass
  • Runtime-control failures

WHY TEST

Why should organizations red team AI systems?

AI systems can behave in ways that are difficult to predict from static code review alone.

Probabilistic model behaviorNatural-language instruction handlingUntrusted contextExternal toolsRetrieval systemsDynamic planningAutonomous actionsChanging models and promptsThird-party components

Red teaming helps answer: “What could this AI system do under adversarial conditions?”

SECURITY DISCIPLINES

AI red teaming vs. penetration testing

The two disciplines overlap. AI red teaming does not replace traditional penetration testing.

PENETRATION TESTINGSoftware, infrastructure, authentication, authorization, network exposure, and application flaws.
AI RED TEAMINGModel behavior, prompt injection, context, tools, agent autonomy, AI-specific data exposure, and multi-step behavior.

EXPECTED VS. UNEXPECTED

AI red teaming vs. model evaluation

MODEL EVALUATIONMeasures quality, accuracy, reliability, safety, and task performance.
RED TEAMINGSearches for failure modes, unsafe behavior, attack paths, control bypass, and exploitable combinations.

Evaluation measures expected behavior. Red teaming searches for unexpected behavior.

MODEL VS. SYSTEM

AI red teaming vs. LLM red teaming

LLM red teaming tests the model layer. AI red teaming can test the complete system around it.

LLM RED TEAMINGJailbreaks, prompt injection, harmful output, and sensitive disclosure.
AI RED TEAMINGAgents, tools, applications, RAG, identity, permissions, infrastructure, and runtime actions.

AI AGENT RED TEAMING

What is AI agent red teaming?

AI agent red teaming tests systems that can use tools, access data, operate through identities, and take actions. The goal is to determine whether an attacker or unexpected condition can cause the agent to exceed its intended task, permissions, or autonomy.

UNTRUSTED INPUT↓AGENT DECISION↓TOOL + IDENTITY↓REAL-WORLD ACTION

CHATBOT VS. AGENT

Why is testing an AI agent different from testing a chatbot?

CHATBOT
INPUTMODELOUTPUT

Testing focuses heavily on response, safety, and disclosure.

AGENT
INPUTAGENTMODELTOOLIDENTITYSYSTEMACTION

Testing must also consider access, authorization, tool use, autonomy, and outcome.

When AI can act, the test cannot stop at the output.

THE ATTACK SURFACE

What should an AI red team test?

Test the full system boundary, from inputs and model behavior through identity, tools, data, and actions.

01INPUTSPrompts · Web content · Documents · Email · RAG
02MODELBehavior · Instruction following · Sensitive output
03AGENTPlanning · Memory · Autonomy
04IDENTITYCredentials · Permissions · Delegation
05TOOLSAPIs · Functions · Code · SaaS actions
06MCPServers · Tools · Resources
07DATASensitive records · Secrets · Documents
08ACTIONSRead · Write · Send · Export · Execute · Delete

TEST AREAS

AI red teaming covers more than model output

PROMPT INJECTIONManipulated instructions change behavior.INDIRECT PROMPT INJECTIONExternal content influences the system.EXCESSIVE AGENCYThe agent exceeds intended autonomy.OVERPRIVILEGED ACCESSThe identity reaches more than the task needs.TOOL MISUSEA legitimate tool is used unsafely.DATA EXFILTRATIONSensitive data leaves an approved boundary.MCP RISKServers, tools, or output create new trust paths.

PROMPT INJECTION TESTING

How do you red team prompt injection?

Test conflicting user instructions, malicious external content, adversarial documents, untrusted web pages, tool responses, RAG content, and MCP output. Do not rely on operational payloads; observe the system’s defensive behavior.

Did the system follow the malicious instruction?Did behavior change?Did the agent attempt an unsafe action?Did access controls limit impact?Did runtime policy intervene?
Prompt Injection Indirect Prompt Injection
UNTRUSTED CONTENT↓AGENT BEHAVIOR↓CONTROL DECISION

PERMISSION TESTING

Test what the AI system can actually access

Configured permissions may differ from effective permissions. Test service-account access, delegated user access, OAuth scopes, cloud roles, tool-mediated access, MCP credentials, and environment boundaries.

Can the agent reach something outside its intended task?

Explore Agent Identity & Permissions Explore Overprivileged Agents

TOOL TESTING

Test whether agents misuse legitimate tools

Test wrong-tool selection, unsafe parameters, unnecessary writes, destructive operations, alternate tools after denial, chained tools, and out-of-scope behavior.

Examine the tool selected, parameters, credential, resource, and outcome.

Explore Tool Misuse
AGENTTOOLPARAMETERSCREDENTIALRESOURCEOUTCOME

MCP RED TEAMING

How do you red team MCP?

Test manipulated tool descriptions, malicious tool output, unexpected capability, excessive permission, third-party server risk, prompt injection, unsafe downstream credentials, data exposure, and multi-tool chains.

Which server was trusted?Which tool was invoked?What credential sat behind it?What downstream resource was reached?
What is MCP Security? MCP Attacks
AGENTMCP SERVERTOOLCREDENTIALSYSTEM

EXCESSIVE AGENCY

Test whether agents exceed their intended authority

Test actions beyond task, high-impact actions without approval, chains that bypass review, retries after denial, delegated authority, and alternate paths.

Explore Excessive Agency
PRIMARY PATH DENIED↘ALTERNATE TOOL↘SAME PROHIBITED OUTCOME?

DATA EXFILTRATION TESTING

Test whether sensitive data can leave trusted boundaries

Test exports, external messages, uploads, third-party APIs, MCP tools, agent-to-agent transfer, and multi-step data movement.

What data was accessed?Which identity enabled it?Where did the data go?Was the destination approved?Did runtime control stop it?
Explore Data Exfiltration

SEQUENCE RISK

Agentic red teaming should test the sequence

Individual actions may appear acceptable. The full chain may not.

01Agent reads malicious document
02Agent queries internal data
03Agent generates file
04Agent invokes external upload
05Sensitive data leaves boundary

Agentic security failures often emerge across multiple legitimate steps.

HUMAN APPROVAL

Test whether approval controls can be bypassed

Test split high-impact actions, retries after denial, alternate tools, changed parameters, delegation to another agent, and context changes.

Can the agent reach the same prohibited outcome through another path?

01High-impact action
02Approval requested
03Approval denied
04Agent retries
05Alternate path attempted
06Outcome observed

RUNTIME CONTROL

Validate allow, review, and block behavior

Red teaming should verify the correct policy decision, safe failure behavior, approval enforcement, and auditability.

Explore Runtime Security
EXPECTED ACTIONRead approved dataALLOW
SENSITIVE ACTIONExternal transferREVIEW
PROHIBITED ACTIONDelete production resourceBLOCK

FAILURE MODES

What happens when a security control fails?

Policy engine unavailableIdentity context missingApproval service unavailableTool metadata incompleteLogging failureTimeoutPartial workflow failure

Define whether the system fails open, fails closed, falls back to review, or pauses the action. The right choice depends on the system and risk.

CHANGE-DRIVEN TESTING

When should AI systems be red teamed again?

Model changesSystem prompt changesTools addedMCP server addedPermissions changeData source changesWorkflow changesAgent autonomy increasesOwnership changesMove to production

AI systems evolve faster than a one-time security assessment.

TESTING LIFECYCLE

A practical AI red teaming lifecycle

01SCOPEDefine the system and trust boundaries.
02MODELUnderstand components and possible actions.
03THREAT MODELIdentify realistic attack paths.
04TESTRun adversarial scenarios.
05OBSERVECapture behavior and outcomes.
06REPORTDescribe the finding and impact.
07REMEDIATEReduce access or strengthen controls.
08RETESTVerify the mitigation.
09MONITORRepeat as the system changes.

REPORTING

What should an AI red-team finding include?

Good findings make the behavior, path, authority, impact, and control gap understandable and reproducible where safe.

FINDINGWhat went wrong?
ATTACK PATHHow did the behavior occur?
ASSETWhich agent or system was affected?
IDENTITYWhich credential or role was involved?
ACTIONWhat did the system attempt?
IMPACTWhat could happen?
CONTROL FAILUREWhich boundary failed?
RECOMMENDATIONHow can risk be reduced?
EVIDENCEWhat supports the finding?

ILLUSTRATIVE EXAMPLE

Example finding: indirect prompt injection triggers a sensitive tool request

PATHExternal document → Agent → MCP tool → CRM write
IDENTITYsvc-agent
IMPACTUnauthorized record modification
CONTROL GAPWrite action does not require approval
RECOMMENDATIONReduce tool permission and add approval for sensitive writes

Illustrative example. This does not describe an actual Eyrane finding.

PRIORITIZATION

How should AI red-team findings be prioritized?

Consider sensitive data, permission level, action impact, exploitability, autonomy, external exposure, environment, blast radius, repeatability, and control coverage.

Use organizational risk judgment rather than an invented universal score.

Sensitive dataPermission levelAction impactExploitabilityAutonomyExternal exposureEnvironmentBlast radiusRepeatabilityControl coverage

PRACTICAL GUIDANCE

AI red teaming best practices

01TEST THE FULL SYSTEM
02DEFINE TRUST BOUNDARIES
03INCLUDE IDENTITY
04TEST UNTRUSTED CONTENT
05TEST TOOLS
06INCLUDE MCP
07TEST MULTI-STEP PATHS
08VALIDATE APPROVAL
09TEST FAILURE MODES
10RECORD EVIDENCE
11RETEST AFTER REMEDIATION
12REPEAT WHEN SYSTEMS CHANGE

SECURITY CHECKLIST

AI red teaming checklist

AUTOMATION + HUMAN REASONING

Can AI red teaming be automated?

Automation can help with repeated scenarios, broad test coverage, regression testing, and standardized attack cases.

Human testers remain valuable for creative attack paths, system-level reasoning, business context, chained vulnerabilities, and novel misuse cases.

Automation increases coverage; human reasoning often improves depth.

AUTOMATIONCoverage · Repeatability · Regression
HUMAN TESTINGCreativity · Context · Novel paths

CONTINUOUS SECURITY

Testing and runtime monitoring complement each other

Red teaming finds weaknesses. Runtime security controls what happens during operation. Monitoring observes behavior. Investigation explains unexpected events.

RED TEAMIMPROVE CONTROLDEPLOYMONITORLEARNRETEST

AUTHORITATIVE GUIDANCE

AI red teaming frameworks and guidance

These resources provide shared language for AI risk, adversarial techniques, and secure agentic applications. They do not imply endorsement of Eyrane.

NIST AI RMFAI risk-management and generative AI guidance.OWASPAI and agentic security threats and mitigations.MITRE ATLASAdversarial tactics and techniques for AI systems.

EYRANE’S POINT OF VIEW

Test from instruction to action

Agentic systems expand the scope of AI red teaming.

A useful test should follow the path from input to model behavior, identity, tool use, downstream access, and final action.

Eyrane is focused on that complete security path.

INPUTAGENTIDENTITYTOOLACTION
Explore AI Red Teaming

COMMON QUESTIONS

Questions about AI red teaming

What is AI red teaming?

AI red teaming is the practice of deliberately testing AI systems with adversarial inputs, attack scenarios, and misuse cases to identify security weaknesses, unsafe behavior, and control failures.

Why is AI red teaming important?

AI red teaming helps organizations find failure modes that static code review and expected-case evaluation may miss, especially when systems process untrusted content, use tools, or take actions.

How is AI red teaming different from penetration testing?

Penetration testing typically focuses on software, infrastructure, authentication, authorization, network exposure, and application flaws. AI red teaming adds model behavior, prompt injection, context, tools, autonomy, and multi-step AI behavior.

How is AI red teaming different from model evaluation?

Evaluation measures expected quality, accuracy, reliability, safety, or task performance. Red teaming intentionally searches for unexpected behavior, attack paths, control bypasses, and exploitable combinations.

What is LLM red teaming?

LLM red teaming tests the model layer for jailbreaks, prompt injection, harmful output, sensitive disclosure, and other model or application safety failures.

What is AI agent red teaming?

AI agent red teaming tests systems that use tools, access data, operate through identities, and take actions, including whether an unexpected condition can exceed intended task, permission, or autonomy.

How do you test prompt injection?

Test conflicting instructions, malicious external content, adversarial documents, web pages, tool responses, RAG content, and MCP output, then observe behavior, attempted actions, access controls, and runtime policy.

How do you red team MCP?

Test tool descriptions, tool output, unexpected capabilities, permissions, third-party server risk, prompt injection, downstream credentials, data exposure, and multi-tool chains.

How do you test AI agent permissions?

Map configured and effective permissions across service accounts, delegated users, OAuth scopes, cloud roles, tools, MCP credentials, and environment boundaries, then test whether the agent can reach outside its task.

What is excessive agency testing?

Excessive agency testing checks whether an agent performs actions beyond its task, bypasses approval, retries after denial, delegates excessive authority, or finds an alternate path.

Can AI red teaming test data exfiltration?

Yes. Testing can examine exports, external messages, uploads, third-party APIs, MCP tools, agent-to-agent transfer, and multi-step data movement.

Should AI red teaming be continuous?

AI red teaming should be repeated as models, prompts, tools, MCP servers, permissions, data sources, workflows, autonomy, ownership, or deployment environments change.

Can AI red teaming be automated?

Automation helps with repeated scenarios, broad coverage, regression testing, and standardized cases. Human testers add creative attack paths, system reasoning, business context, and novel misuse cases.

Who performs AI red teaming?

AI red teaming may involve security engineering, AppSec, AI platform teams, red teams, privacy, data security, engineering, GRC, and specialized external testers.

When should an AI system be red teamed?

Test before production and retest after material changes to models, prompts, tools, permissions, data, workflows, autonomy, or trust boundaries.

RELATED LEARNING

Continue learning

What is AI Security?What is AI Agent Security?What is Agentic Security?What is MCP Security?What is AI Agent Identity?

RELATED PLATFORM

Explore AI security testing

AI Red TeamingAI Agent SecurityRuntime SecurityAgent Identity & PermissionsMCP Security

RELATED THREATS

Explore AI attack paths

Prompt InjectionIndirect Prompt InjectionExcessive AgencyOverprivileged AgentsTool MisuseData ExfiltrationMCP Attacks

AI RED TEAMING

Test what happens when AI can act.

Evaluate prompts, permissions, tools, MCP connections, data access, approval controls, and multi-step behavior before unsafe paths reach production.

Explore AI Red TeamingGet Early Access