How to Test AI Agents for Prompt Injection

Prompt-injection testing is incomplete when it stops at the model response. The escape that matters is the one that changes a tool call, recipient, record, or other downstream target. This guide builds a test method that can observe that full path.

A prompt-injection test case needs an oracle before it needs more payloads. Record the fixture channel, expected decision, tool call that would prove escape, and a disposable object whose unchanged state proves the denial worked.

Use the MCP protocol architecture reference only if a test calls an MCP tool; it helps name the client and server evidence in the run.

Define the protected action and the test oracle

Choose one action that matters, such as sending a message, reading a secret, changing a record, or approving a payment. Define the exact allowed state change.

The harness can prove both the model output and the downstream state for each case. Do not start broad red teaming while the action has no observable success or failure signal.

Build a corpus by attack channel

Include direct user instructions, poisoned documents, tool-output instructions, HTML or markdown payloads, encoded text, and multi-turn persistence cases.

Every fixture records its source channel, expected classification, tool availability, and target state. Remove production secrets and live recipients from fixtures; use disposable tenants and harmless targets.

Use synthetic test identities rather than a copied developer token. The agent credential leak controls guide is relevant here because a test transcript that captures a bearer value becomes a separate security failure.

Exercise the full agent path

Run each case through retrieval, planning, tool selection, policy enforcement, and execution. A model-only chat test cannot prove runtime safety.

A denied case neither invokes a side-effecting tool nor changes the test target. Stop a test run when the harness can call a production connector without an isolation boundary.

Score failures by capability gained

Prioritize cases that change authorization, redirect a tool, expose data, or persist instructions over harmless format changes.

The report distinguishes model compliance, attempted tool call, policy denial, and completed effect. Quarantine a release when a new capability is obtained without the expected approval or scope.

Turn regressions into a release gate

Keep confirmed cases in CI or a scheduled evaluation suite. Re-run after prompt, model, retrieval, tool, policy, or connector changes.

A release requires a passing baseline, no unresolved high-impact escape, and retained evidence for each exception. Roll back the changed policy, tool exposure, or prompt and verify the regression before reopening the release.

Make the regression result a release decision

Archive the minimized regression fixture with its expected result and observed side effect. That lets a later model or prompt change be tested against the same attack rather than a vague recollection of the issue.

Use OWASP Prompt Injection and OpenAI safety evaluations guidance to broaden the attack corpus, then keep only cases whose expected tool decision and target-state assertion are explicit.

A Deep Scan finding can seed a regression only when it identifies the exact component and version under test. Turn the finding into a controlled fixture, then judge the run by the normalized tool call and harmless target state. For each seeded case, record the delivery channel, the authority change it attempts, and the downstream effect that would prove escape.

The component coverage in AgentGuard documentation determines whether that finding belongs in this corpus; it never becomes the test oracle.