AI Privacy: Trace the Data Before You Trust the Answer

AI privacy is easy to discuss at the policy level and hard to verify in a working system. A user prompt may pass through a browser, application logs, a retrieval service, a model provider, connected tools, and an analytics pipeline. Each copy can have a different owner and retention rule. The first privacy control is therefore a data-flow record that reflects the deployed path, not the architecture diagram from procurement.

Map every copy of personal data

Trace inputs from collection to deletion. Include prompt text, uploaded files, embeddings, retrieval indexes, model requests, tool arguments, traces, support exports, and evaluation datasets. Record purpose, lawful basis where applicable, owner, region, retention, encryption, and deletion mechanism for each store. A field removed from the chat interface may still exist in logs or an evaluation corpus.

The Model Context Protocol can expose tools and resources to an agent. Its presence does not determine privacy. Review what a specific server can read, what identity it uses, and whether its responses are copied into model context or telemetry.

Minimize before redacting

Do not send a full record when the task needs two fields. Minimize at the application boundary, then apply deterministic redaction for known identifiers before data reaches the model. Redaction is not a substitute for access control: a user who may not read a record should not be allowed to ask an agent to summarize it.

Stanford HAI's discussion of privacy in the AI era explains why inference and aggregation complicate traditional notice-and-consent approaches. The operational response is to reduce the data available to each task and retain fewer secondary copies.

Control retrieval and tool output

Retrieval can turn a narrow prompt into a broad disclosure if the index ignores tenant, role, or document-level permissions. Enforce authorization before retrieval and again before a tool changes state. Label provenance so the output can be traced to the source record. Test with two users from different scopes and confirm that the same query returns different, authorized results.

Treat retrieved text as untrusted as well as sensitive. The prompt injection path matters because a document can attempt to redirect the agent toward another file, recipient, or tool.

Set retention and deletion by artifact

A single '30-day retention' statement is not enough. Prompts, traces, embeddings, backups, fine-tuning datasets, and security evidence may follow different schedules. Define deletion for each artifact and test it with a synthetic identifier that can be searched across stores. Preserve only the minimum security evidence needed to investigate an event.

The NIST Privacy Framework provides a structure for identifying and managing privacy risk. Use it to assign decisions, then verify the implementation with data-store and deletion tests.

Review privacy when the agent gains authority

Privacy risk changes when a chatbot becomes an agent. A tool can export a report, send an email, update a CRM, or join data that was previously separated. Re-run the privacy assessment after adding a connector, expanding a service account, or changing the destination policy. Track the final recipient and target state in tests.

Use AI agent threat modeling to connect data paths to abuse cases. AgentGuard can support component and runtime reviews, but teams must still define which identities, records, and destinations are allowed. Map a privacy-sensitive agent path before adding a new high-impact tool.

Before approval, reproduce one complete privacy request: identify the source record, show the authorization decision, locate every retained copy, execute deletion, and confirm the identifier is no longer retrievable. This exercise catches ownership gaps that a policy review misses. Record exceptions explicitly, including the reason, owner, expiry, and the store that cannot yet meet the target.