Home / Knowledge hub / AI data leakage
Knowledge hub
AI data leakage
A model or its logs exposing private data that went into prompts or training.
What it means
AI data leakage is private or sensitive information escaping through an AI system. That includes a model repeating something from its training data, a chat history shown to the wrong person, prompts and answers kept in logs, and staff pasting confidential material into an outside tool that stores or learns from it.
The idea that makes it click: an AI feature is a new place where data is copied, processed and displayed. Every place the text goes, whether the prompt, the retrieved documents, the logs or the vendor's servers, needs the same protection as the original data.
Why it matters
The consequences are those of any data breach: customer data exposed, contract and privacy law obligations, and loss of trust. An internal assistant that answers from company files can reveal salaries or customer records to people who should not see them, simply because it was asked politely.
Because AI tools are adopted quickly, they often arrive without review. Staff use them before policy exists, and vendors change terms about retaining prompts. Leaked personal data can trigger breach notification duties, and sending personal data to a vendor in another country raises residency questions.
How it shows up
The routes repeat. Prompts or conversation logs stored without limits or access control. Shared chat links that anyone can open. An assistant that retrieves documents without checking the asker's permissions, see RAG security. Hidden instructions in a web page or file that make the model send data out, see indirect prompt injection.
Check what you send to models, what the vendor retains and for how long, who can read stored conversations, and whether each answer respects the asker's access. Keep secrets and personal data out of prompts where you can, and apply data loss prevention to tools staff use. OWASP lists sensitive information disclosure among the top risks for LLM applications.
Questions people ask
AI data leakage, answered
Can a model leak its training data?
Yes, in some cases. Models can memorise and repeat rare strings from their training data, especially if it contained duplicates or secrets.
Keep sensitive data out of training sets and test for it.
Is pasting company data into a chatbot a leak?
It can be. The data leaves your control and may be stored, reviewed or used for training depending on the service and its terms.
Set a policy, offer an approved tool with suitable contract terms, and tell staff what must never be pasted.
How is AI data leakage different from prompt injection?
Prompt injection is a way to make a model follow hostile instructions. Data leakage is the result when private information is exposed, which injection is one of several ways to cause.
Weak permission checks can leak data with no injection at all.
Do AI vendors keep our prompts?
Some do for a period, for abuse monitoring or improvement, depending on plan and settings. Read the data processing terms and the retention settings.
Enterprise plans often offer stronger commitments, but confirm in writing.
How can we test our AI feature for leakage?
Ask it for data belonging to other accounts, check stored conversations for access control, review what retrieved documents it can cite, and look at the logs.
Do this with accounts of different permission levels, not one test login.
Sources
Where this comes from
Related
Terms that sit next to this one
RAG security
Controlling what a retrieval-based AI assistant can fetch and show to each person.
Prompt injection
Text that tricks an AI model into ignoring its instructions and doing something else.
Indirect prompt injection
Instructions hidden in a web page or file that an AI assistant then obeys.
OWASP LLM Top 10
The OWASP list of the main security risks in applications built on language models.
Data loss prevention
Tools and rules that stop sensitive data leaving where it should stay.
Who else can your assistant answer for?
We check business-logic flaws and cross-account access, including whether an AI feature returns another account's data, and every finding comes with the request and response that proved it. Book a pilot on one app.
Or start with a $199 pilot on one application: thirty days, success criteria agreed before day one, credited against the annual if you convert.