AI systems should be tested with more than expected user requests. Red teaming deliberately explores how a model, retrieval layer, agent or connected tool can be manipulated, confused or pushed beyond policy. The purpose is not to produce dramatic attack transcripts; it is to discover failure paths and convert them into controls and repeatable tests.
This guide treats adversarially testing AI systems before launch as an operating system rather than a feature. The useful question is whether people can use it consistently, observe the outcome, handle exceptions and improve the process without creating hidden risk or unnecessary complexity.
What AI red teaming means
AI red teaming is structured adversarial evaluation performed by authorized testers. It covers misuse, prompt injection, data leakage, unsafe tool actions, identity and permission weaknesses, harmful outputs, model limitations and the surrounding application architecture.
Agents expand the potential impact because an unsafe output can become an action. NIST’s current work highlights agent hijacking and security evaluation. Effective red teaming therefore includes tool boundaries, credentials, approvals and recovery—not only model conversation.
Five design principles
Define the system and authorization boundary must become a visible rule, owner and acceptance test. Define a normal case, a difficult case and an unacceptable failure. Record the evidence a reviewer needs so the principle can guide real decisions rather than remaining an attractive phrase.
2. Prioritize plausible high-impact failures
Prioritize plausible high-impact failures must become a visible rule, owner and acceptance test. Define a normal case, a difficult case and an unacceptable failure. Record the evidence a reviewer needs so the principle can guide real decisions rather than remaining an attractive phrase.
3. Test the complete action chain
Test the complete action chain must become a visible rule, owner and acceptance test. Define a normal case, a difficult case and an unacceptable failure. Record the evidence a reviewer needs so the principle can guide real decisions rather than remaining an attractive phrase.
4. Preserve evidence and reproduce findings
Preserve evidence and reproduce findings must become a visible rule, owner and acceptance test. Define a normal case, a difficult case and an unacceptable failure. Record the evidence a reviewer needs so the principle can guide real decisions rather than remaining an attractive phrase.
5. Convert discoveries into regression tests
Convert discoveries into regression tests must become a visible rule, owner and acceptance test. Define a normal case, a difficult case and an unacceptable failure. Record the evidence a reviewer needs so the principle can guide real decisions rather than remaining an attractive phrase.
Implementation workflow
1. Map components, data and tools
Complete this step with a named owner and saved output. Use representative work rather than invented examples, and note unresolved assumptions. Before moving forward, confirm the effect on users, data, cost, control and the manual fallback.
2. Create a threat model
Complete this step with a named owner and saved output. Use representative work rather than invented examples, and note unresolved assumptions. Before moving forward, confirm the effect on users, data, cost, control and the manual fallback.
3. Define rules of engagement
Complete this step with a named owner and saved output. Use representative work rather than invented examples, and note unresolved assumptions. Before moving forward, confirm the effect on users, data, cost, control and the manual fallback.
4. Build normal and adversarial cases
Complete this step with a named owner and saved output. Use representative work rather than invented examples, and note unresolved assumptions. Before moving forward, confirm the effect on users, data, cost, control and the manual fallback.
5. Run controlled attacks
Complete this step with a named owner and saved output. Use representative work rather than invented examples, and note unresolved assumptions. Before moving forward, confirm the effect on users, data, cost, control and the manual fallback.
6. Triage findings by impact
Complete this step with a named owner and saved output. Use representative work rather than invented examples, and note unresolved assumptions. Before moving forward, confirm the effect on users, data, cost, control and the manual fallback.
7. Fix, retest and monitor
Complete this step with a named owner and saved output. Use representative work rather than invented examples, and note unresolved assumptions. Before moving forward, confirm the effect on users, data, cost, control and the manual fallback.
Worked example
A research agent can read websites and save notes. Testers place malicious instructions in a retrieved page, request access to unrelated files and attempt to redirect uploads. The system should treat page text as data, restrict file scope and block unapproved destinations. Every successful bypass becomes a permanent test.
The example works because the scope and feedback loop are explicit. Exceptions do not disappear into private messages. They become evidence for a better rule, stronger test, clearer training or a decision to keep part of the workflow manual.
Metrics and review cadence
Track critical findings, attack success rate, time to contain, retest pass rate, evaluation coverage. Review leading indicators weekly during a pilot and business outcomes monthly. Segment results by user group, case type and risk level because a healthy average can conceal one important class of failure.
- Define each metric in plain language and name its source.
- Compare results with a pre-change baseline.
- Pair speed or volume with quality and risk.
- Record why targets were missed and which change will be tested.
- Retire measures that no longer influence a decision.
Common mistakes
Testing only the base model
This mistake appears when speed is rewarded before the operating conditions are clear. Correct it by narrowing the scope, documenting the assumption, testing a difficult real case and assigning someone to verify the result.
This mistake appears when speed is rewarded before the operating conditions are clear. Correct it by narrowing the scope, documenting the assumption, testing a difficult real case and assigning someone to verify the result.
Reporting findings without reproducible cases
This mistake appears when speed is rewarded before the operating conditions are clear. Correct it by narrowing the scope, documenting the assumption, testing a difficult real case and assigning someone to verify the result.
Fixing prompts while leaving tool permissions broad
This mistake appears when speed is rewarded before the operating conditions are clear. Correct it by narrowing the scope, documenting the assumption, testing a difficult real case and assigning someone to verify the result.
A practical 30-day plan
- Week 1: document the current workflow, intended outcome, baseline and unacceptable failures.
- Week 2: design the smallest controlled version and prepare normal, difficult and exception tests.
- Week 3: run a limited pilot with daily observation, a fallback and a shared issue log.
- Week 4: fix recurring causes, compare results with the baseline and decide whether to expand, redesign or stop.
Connect this work with the prompt injection guide. The surrounding process, roles and measurements determine whether the focused system creates lasting value.
Questions before scaling
- Who owns the business outcome and daily operation?
- Which decisions, data or promises require explicit approval?
- What does a correct result look like in normal and difficult cases?
- How can a user stop the workflow and reach a responsible person?
- Which costs rise with volume, complexity or exception rate?
- What evidence would cause the team to pause or retire the system?
Final takeaway
Red team the system that actually operates: model, context, identity, tools and people. Use findings to strengthen architecture and continuous evaluation before authority expands.
Start small enough to observe closely, but design the evidence from the beginning. Reliable systems grow from clear boundaries, representative tests, useful measures and honest review—not from adding features before the workflow is understood.
Sources and further reading
