Get Early Access
HomePlatformAI Red Teaming

AI RED TEAMING

Test agents before attackers do

AI agents can access data, use tools, and take actions across enterprise systems. Eyrane helps security teams test how those agents behave when prompts, tools, permissions, MCP connections, and workflows are manipulated.

Get Early AccessExplore Agent Security

THE DEFINITION

What is AI red teaming?

AI red teaming is the practice of testing AI systems with adversarial inputs and attack scenarios to identify unsafe behavior, security weaknesses, and control failures. For AI agents, red teaming also evaluates identities, permissions, tools, MCP connections, data access, and actions the agent can take.

  • Prompt injection resistance
  • Indirect prompt injection
  • Unsafe tool use
  • Excessive permissions
  • Authorization boundaries
  • MCP trust
  • Sensitive data exposure
  • Human approval controls
  • Multi-step attack paths

THE CORE DIFFERENCE

Red teaming agents is different from testing chatbots

CHATBOT TESTINGGenerated content · jailbreaks · harmful responses · hallucinations
AGENT TESTINGTool invocation · credential use · authorization · data movement · multi-step behavior · downstream actions

When AI can act, the test cannot stop at the response.

ATTACK SURFACE

Test the full agent attack surface

Agent red teaming follows the path from adversarial input to identity, capability, data, and action.

INPUTSDirect prompts · external content · documents · web pages · tool responses
AGENTReasoning · planning · memory · autonomy
IDENTITYCredentials · roles · permissions · delegated access
TOOLSAPIs · functions · code execution · SaaS actions
MCPServers · tool definitions · tool responses · credentials
DATASensitive inputs · internal documents · records · secrets
ACTIONSRead · write · send · export · delete · execute

PROMPT INJECTION TESTING

Test whether malicious instructions change agent behavior

The test should examine not only whether an injection succeeds conceptually, but whether it leads to an unsafe downstream action.

External page↓Hidden instruction↓Agent processes page↓Sensitive action attempted↓Control evaluates
Prompt Injection Indirect Prompt Injection
TEST QUESTIONDid untrusted content become an unsafe action?OUTCOME: SAFE FAILURE

TOOL MISUSE TESTING

Test what happens when agents use tools incorrectly

Test wrong tools, dangerous parameters, unexpected chains, write capabilities during read-only tasks, and retries after denial.

Explore Tool Misuse
EXPECTED TASKLook up customer status
UNEXPECTED BEHAVIORExport customer recordsIllustrative scenario

PERMISSION TESTING

Test what agents can access, not just what they should access

Compare intended access with effective access across OAuth scopes, service accounts, cloud roles, delegated permissions, stale credentials, and tool-mediated paths.

Agent Identity & Permissions Overprivileged Agents
INTENDED ACCESSRead customer statusversusEFFECTIVE ACCESSExport customer records · write to CRM · post externally

EXCESSIVE AGENCY TESTING

Test whether agents take actions without enough oversight

Permission alone does not determine safety. Red teaming should test autonomy, scope expansion, retries after denial, escalation, and actions outside the intended task boundary.

Explore Excessive Agency
PROCUREMENT AGENT
Requested taskCompare vendors
Unexpected actionSubmit purchase order
Task scope expanded into transaction execution.

MCP RED TEAMING

Test trust across MCP servers and tools

Test malicious tool descriptions, manipulated responses, unexpected capabilities, compromised servers, untrusted dependencies, hidden side effects, and sensitive data exposure.

Explore MCP Security
Agent↓MCP server↓Tool response includes malicious instruction↓Privileged tool selected↓Sensitive action attempted

DATA EXFILTRATION TESTING

Test whether sensitive data can leave trusted boundaries

Each step may be legitimate independently. The chain may not be.

Explore Data Exfiltration
01Query internal database
02Generate report
03Upload report externally
Combined path requires review

MULTI-STEP ATTACK PATHS

Test the sequence, not just individual actions

Agentic risk often emerges from the chain. A document can influence a plan, a tool can expose data, and a later tool can move it outside the trusted boundary.

01Read malicious documentLOW RISK
02Document influences planLOW RISK
03Invoke internal toolLOW RISK
04Expose sensitive dataMODERATE
05Message external destinationHIGH RISK
Single actions may appear safe · combined sequence is unsafe

HUMAN APPROVAL TESTING

Verify that sensitive actions actually require approval

Conceptual tests can ask whether an agent can bypass approval, split an action, use another tool, retry after denial, or act after approval expires.

CONTROLRefunds above $10,000 require approval
ATTACKAgent attempts $9,999 twice
FINDING · Potential policy bypass

RUNTIME CONTROL VALIDATION

Test whether runtime controls fail safely

Red teaming can validate that expected actions work, sensitive actions trigger approval, prohibited actions stop, and unavailable policy context has a defined safe behavior.

Explore Runtime Security
ALLOWExpected low-risk actions work.
REVIEWSensitive actions trigger approval.
BLOCKProhibited actions stop.
FAILURE MODEUnavailable context has a safe behavior.

TEST BEFORE PRODUCTION

Test agents before they reach production

Security testing can span the agent lifecycle.

01BUILDEvaluate intended functionality.
02TESTRun adversarial scenarios.
03REVIEWInspect failures and access paths.
04REMEDIATEReduce permissions or strengthen controls.
05DEPLOYMove to production.
06CONTINUOUS TESTINGRetest when the system changes.

CHANGE-DRIVEN TESTING

Agent risk changes when the system changes

Retesting may be needed when models, prompts, tools, MCP servers, permissions, credentials, data sources, workflows, or autonomy change.

Model changesPrompt changesNew toolsMCP changesPermission changesCredential changesData source changesWorkflow changesMore autonomy

ILLUSTRATIVE FINDING

Make findings actionable

Useful findings connect a behavior to the identity, permission, tool, resource, action, and control that failed.

FINDINGIndirect prompt injection can trigger privileged tool call
SEVERITY HighIDENTITY svc-agent-prod
ATTACK PATH

External content → Agent → MCP tool → Production CRM

IMPACT

Customer data modification

RECOMMENDATION

Reduce tool permissions and require approval for write actions.

RED TEAMING + IDENTITY

Tie every finding back to access

Actionable testing should answer which agent was affected, which identity enabled the behavior, which tool was invoked, which resource was reached, and what control failed.

Explore Agent Identity & Permissions
Agent→Identity→Permission→Tool→Resource

RED TEAMING + RUNTIME

Use testing to improve runtime policy

01RED TEAMFind unsafe path
02UNDERSTANDIdentify identity, tool, and data
03CONTROLAdd or refine policy
04RETESTVerify the control
05MONITORObserve behavior

SECURITY FRAMEWORKS

Test against recognized AI security risks

Frameworks can provide shared language for adversarial testing. Eyrane’s approach connects those risks to agents, permissions, tools, data, and actions.

OWASP Agentic Top 10 MITRE ATLAS AI Red Teaming guide Agentic Security guide

COMMON QUESTIONS

Questions security teams ask about AI red teaming

What is AI red teaming?

AI red teaming is the practice of testing AI systems with adversarial inputs and attack scenarios to identify unsafe behavior, security weaknesses, and control failures.

What is AI agent red teaming?

AI agent red teaming tests not only model responses, but also the identities, permissions, tools, MCP connections, data access, and actions an agent can take.

How is agent red teaming different from LLM red teaming?

LLM red teaming often focuses on jailbreaks and model outputs. Agent red teaming also tests authorization, tool invocation, downstream systems, autonomy, and multi-step behavior.

What security risks should AI agents be tested for?

AI agents should be tested for prompt injection, unsafe tool use, excessive permissions, MCP risks, data exposure, approval bypasses, and dangerous action chains.

Can red teaming detect prompt injection?

Red teaming can test whether direct or indirect prompt injection changes agent behavior and whether that influence leads to an unsafe downstream action.

How do you test AI agent permissions?

Permission testing compares intended access with effective access across credentials, roles, delegated permissions, tools, and connected systems.

How do you red team MCP servers?

MCP red teaming tests tool descriptions, tool responses, server trust, downstream permissions, hidden side effects, and sensitive actions exposed through MCP.

What is tool misuse in AI agents?

Tool misuse is when an agent invokes a legitimate tool in an unsafe, unauthorized, or unintended way, including with dangerous parameters or outside task context.

Can AI red teaming test data exfiltration?

Yes. Testing can trace whether an agent can combine reads, transformations, files, and external tools into a path that moves sensitive data outside an approved boundary.

Why are multi-step attacks important for AI agents?

Multi-step attacks matter because individually reasonable actions can combine into an unsafe sequence that a single-call test would miss.

Should AI agents be red teamed continuously?

Agents should be retested when models, prompts, tools, MCP servers, permissions, credentials, data sources, workflows, or autonomy change.

How does AI red teaming improve runtime security?

Red teaming finds unsafe paths that can inform runtime policy, approval requirements, permissions, and monitoring before deployment.

RELATED CONTENT

AI Agent Security Agent Discovery Agent Identity & Permissions Runtime Security MCP Security Prompt Injection Indirect Prompt Injection Excessive Agency Tool Misuse Data Exfiltration

AI RED TEAMING

Find unsafe agent behavior before production does.

Test prompts, permissions, tools, MCP connections, and multi-step actions before they become security incidents.

Get Early Access