How to Audit an AI System Without Turning It into a Paper Exercise

An AI audit should answer a concrete question: can this system operate inside its approved purpose, data boundary, and authority? A policy inventory is useful, but it is not the audit result. The useful result is evidence that connects a requirement to a system component, a test, and an observed outcome. This guide shows how to build that chain for models, applications, and tool-using agents.

Set the audit question before collecting evidence

Start with the decision the audit must support. A release audit may ask whether a new agent can access customer records and send messages under the right identity. A vendor review may ask whether the provider's controls cover your deployment model. Write the system boundary, intended users, prohibited outcomes, review period, and accountable owner. Without those limits, evidence collection expands indefinitely and still fails to answer whether the system is acceptable.

Use an AI agent threat modeling exercise to locate the trust boundaries that deserve testing. Keep the audit scope tied to deployed components and reachable actions, not every risk that could exist in an abstract model.

Build an evidence map, not a document pile

For each control, record the requirement, owner, implementation point, evidence source, test procedure, result, and expiry date. A screenshot of a settings page proves only what was visible at that moment. A useful evidence item includes the tenant, configuration version, identity, and test target. Logs should show both an allowed case and a denied case so a blanket outage cannot pass as effective enforcement.

Treat statements from a provider as design evidence until your deployment confirms them. The NIST AI Risk Management Framework organizes work around Govern, Map, Measure, and Manage; your evidence map should connect those functions to specific system behavior.

Test the paths that can change state

Prioritize actions that disclose data, modify records, spend money, change permissions, or trigger downstream automation. Run the test through the same identity, retrieval layer, tools, and policy components used in production. Check the final target state, not only the model transcript. An agent can say that it refused while a queued job still executes.

Use MITRE ATLAS to expand adversarial cases when a technique matches the system. Do not turn the framework into a checklist of labels. Select a technique, identify the exposed path, run a bounded test, and retain the observable result.

Separate design gaps from operating failures

A missing approval rule is a design gap. A correct rule that was disabled during an incident is an operating failure. A test that cannot be reproduced is an evidence gap. Keep these categories separate because the owner and remediation differ. Assign severity from reachable impact and exposure, then add an owner, due date, retest method, and temporary restriction.

IBM's overview of AI audits describes governance, data, performance, and compliance concerns. In practice, the audit becomes actionable only when each concern is tied to a deployed component and a verification step.

Close the audit with a release decision

An audit is not closed when the report is delivered. It closes when every material finding has a disposition: fixed and retested, accepted by an accountable owner, mitigated by a temporary restriction, or blocking release. Preserve the exact test fixtures and component versions behind the decision. Set the next review trigger around change: a new model, connector, tool, permission scope, or data source can invalidate the result before the calendar review date.

AgentGuard can help teams examine agent components and runtime control points, but the audit conclusion still depends on the organization's own scope and observed outcomes. Review an agent path with AgentGuard when you need to turn a broad audit finding into a testable control question.

Use the NIST AI RMF glossary to align local evidence with the framework's four functions without replacing the system-specific test record.