Prompt injection happens when untrusted content influences an AI system to ignore intended rules, reveal information or misuse tools. The content may arrive in a user message, webpage, document or retrieved record. Because models process instructions and data together, ordinary text can become an attack path.
This guide treats defending AI systems against malicious instructions as an operating system rather than a one-time project. The useful question is not whether a tool or framework exists. It is whether people can use it consistently, observe the result, handle exceptions and improve the process without creating hidden risk.
What prompt injection means
Prompt injection is an attempt to manipulate a model through crafted input. Direct injection comes from the user; indirect injection is embedded in external content the system reads. Impact depends on the data and actions available to the model.
A perfect system prompt is not a dependable security boundary. Defenses must surround the model with least privilege, content boundaries, tool validation, approvals, output handling and monitoring.
Five design principles
1. Treat retrieved content as untrusted data
Treat retrieved content as untrusted data must be translated into a visible rule, owner and acceptance test. Discuss what a good case looks like, what can go wrong and which evidence a reviewer needs. This turns an attractive idea into a repeatable part of real work.
Separate authority from natural-language claims must be translated into a visible rule, owner and acceptance test. Discuss what a good case looks like, what can go wrong and which evidence a reviewer needs. This turns an attractive idea into a repeatable part of real work.
3. Give tools least privilege
Give tools least privilege must be translated into a visible rule, owner and acceptance test. Discuss what a good case looks like, what can go wrong and which evidence a reviewer needs. This turns an attractive idea into a repeatable part of real work.
4. Validate actions outside the model
Validate actions outside the model must be translated into a visible rule, owner and acceptance test. Discuss what a good case looks like, what can go wrong and which evidence a reviewer needs. This turns an attractive idea into a repeatable part of real work.
5. Require approval for consequential operations
Require approval for consequential operations must be translated into a visible rule, owner and acceptance test. Discuss what a good case looks like, what can go wrong and which evidence a reviewer needs. This turns an attractive idea into a repeatable part of real work.
Implementation workflow
1. Map inputs, data and tools
Complete this step with a named owner and a saved output. Use representative cases rather than invented examples, and record unresolved assumptions. Before moving forward, confirm how the step affects users, data, cost, controls and the manual fallback.
2. Classify potential impact
Complete this step with a named owner and a saved output. Use representative cases rather than invented examples, and record unresolved assumptions. Before moving forward, confirm how the step affects users, data, cost, controls and the manual fallback.
3. Reduce exposed capabilities
Complete this step with a named owner and a saved output. Use representative cases rather than invented examples, and record unresolved assumptions. Before moving forward, confirm how the step affects users, data, cost, controls and the manual fallback.
4. Label and isolate untrusted content
Complete this step with a named owner and a saved output. Use representative cases rather than invented examples, and record unresolved assumptions. Before moving forward, confirm how the step affects users, data, cost, controls and the manual fallback.
5. Validate tool parameters and destinations
Complete this step with a named owner and a saved output. Use representative cases rather than invented examples, and record unresolved assumptions. Before moving forward, confirm how the step affects users, data, cost, controls and the manual fallback.
6. Test direct and indirect attacks
Complete this step with a named owner and a saved output. Use representative cases rather than invented examples, and record unresolved assumptions. Before moving forward, confirm how the step affects users, data, cost, controls and the manual fallback.
7. Monitor denials and suspicious behavior
Complete this step with a named owner and a saved output. Use representative cases rather than invented examples, and record unresolved assumptions. Before moving forward, confirm how the step affects users, data, cost, controls and the manual fallback.
Worked example
An assistant reads a webpage that contains hidden instructions to upload private files. The system treats page text only as evidence, not authority. The file tool is not available for that workflow, outbound destinations are allow-listed and sensitive actions require confirmation. The injection may change the draft text, but it cannot create the prohibited action.
The example works because the scope is narrow and the feedback loop is explicit. Exceptions do not disappear into private messages. They become evidence for better rules, clearer training, stronger tests or a decision to keep part of the workflow manual.
Metrics and review cadence
Track attack test pass rate, blocked tool requests, sensitive-data exposure, approval rejection reasons, time to revoke capability. Review leading indicators weekly during a pilot and business outcomes monthly. Segment results by user group, case type and risk level. Averages can look healthy while one important class of work is failing.
- Define every metric in plain language and name its source.
- Compare results with a pre-change baseline, not only with the previous week.
- Pair speed or volume with a quality and risk measure.
- Record why targets were missed and which change will be tested next.
- Retire metrics that no longer influence a decision.
Common mistakes
Relying only on a stronger system prompt
This mistake usually appears when speed is rewarded before the operating conditions are clear. Correct it by narrowing the scope, documenting the assumption, testing a difficult real case and assigning someone to verify the result.
Letting the model decide its own permissions
This mistake usually appears when speed is rewarded before the operating conditions are clear. Correct it by narrowing the scope, documenting the assumption, testing a difficult real case and assigning someone to verify the result.
Rendering model output as trusted code
This mistake usually appears when speed is rewarded before the operating conditions are clear. Correct it by narrowing the scope, documenting the assumption, testing a difficult real case and assigning someone to verify the result.
Skipping tests with malicious documents
This mistake usually appears when speed is rewarded before the operating conditions are clear. Correct it by narrowing the scope, documenting the assumption, testing a difficult real case and assigning someone to verify the result.
A practical 30-day plan
- Week 1: document the current workflow, outcome, baseline, users and unacceptable failures.
- Week 2: design the smallest controlled version and prepare normal, difficult and exception test cases.
- Week 3: run a limited pilot with daily observation, a manual fallback and a shared issue log.
- Week 4: fix recurring causes, compare results with the baseline and decide whether to expand, redesign or stop.
Connect this work with the AI governance guide. The surrounding process, roles and measurements determine whether the focused system creates lasting value.
Questions before scaling
- Who owns the business outcome and who owns day-to-day operation?
- Which decisions, data or promises require explicit approval?
- What does a correct result look like across normal and difficult cases?
- How will a user stop the workflow and reach a responsible person?
- Which costs rise with volume, complexity or exception rate?
- What evidence would cause the team to pause or retire the system?
Final takeaway
Prompt injection is a system security problem, not a wording puzzle. Assume untrusted content can influence the model and design authority, tools and approvals so manipulated text cannot become uncontrolled action.
Start small enough to observe closely, but design the evidence from the beginning. Reliable systems grow from clear boundaries, representative tests, useful measures and honest review—not from adding more features before the basic workflow is understood.
Sources and further reading
