In Depth
Classic data leakage assumes an attacker gets in: someone steals credentials, exploits a vulnerability, exfiltrates a database. The defenses (network segmentation, access controls, intrusion detection) are all built around that perimeter. An AI agent breaks the model. It can disclose data while every control is working as designed, because the leak path runs straight through the system's intended function: generating output.
The exposure takes several shapes. A model trained or fine-tuned on sensitive records can regurgitate them verbatim to a later user. A retrieval-augmented agent can pull a document one user isn't entitled to see and summarize it into an answer. Prompt logs and trace tooling can quietly persist secrets that were only ever meant to live in memory. And a prompt injection can deliberately coax an agent into dumping its context, its system prompt, or another tenant's data. OWASP catalogs this as Sensitive Information Disclosure in its Top 10 for LLM Applications. It is a named, recognized failure mode, not an edge case.
What makes it hard is the same thing that makes hallucination hard: there is no alarm. The agent returns a clean, helpful answer. Nothing in the logs reads as an incident until someone notices the answer contained data it should never have surfaced.
What It Looks Like
A vendor ships an internal-knowledge agent to a customer. Two business units share the same deployment but are supposed to be walled off from each other. A salesperson asks the agent a routine question, and the retrieval layer, indexed across both units without tenant filtering, returns and summarizes a document from the other team's confidential pipeline. No one broke in. No malware ran. The agent did precisely what it was built to do, and in doing so disclosed data across a boundary that was supposed to hold.
Why It Matters For AI Vendors
Your enterprise customer's security team will ask about data isolation, retention, and leakage in their review, and they will not accept "our cyber policy covers it" as an answer, because increasingly it does not. Cyber insurance responds to a breach of systems. A model that discloses data through normal operation, with no intrusion, can fall outside that grant. The vendor is left holding a contractual confidentiality obligation with no policy clearly behind it.
When the disclosed data is regulated, whether that's health records, financial data, or personal information, the disclosure also triggers statutory and notification duties on top of the contract. One leak becomes a legal event, a customer-relationship event, and a coverage question all at once.