Score sensitive-information disclosure risk in LLM systems: training data, retrieval scope, tenant isolation, logging and memorisation.
Most real LLM data leakage is not exotic model extraction — it is retrieval returning documents the user was never entitled to see, because embeddings were indexed without permission metadata. Prompt logs are the second common route: a general-purpose log store quietly accumulates every secret anyone pasted into a chat window. Memorisation matters mainly for small, repeated fine-tuning sets. The score is a prioritisation estimate for design review, not assurance. A retrieval layer that ignores the user's permissions turns a chatbot into a search engine for everything the service account can read, which is usually everything.
LLM Data Leakage Risk
likelihood = mean(retrieval scope, redaction gap, memorisation risk, output filtering gap); impact = mean(training-data sensitivity, prompt-retention risk) plus a record-volume uplift; risk = likelihood × impact ÷ 10.
likelihood = mean(retrieval scope, redaction gap, memorisation risk, output filtering gap); impact = mean(training-data sensitivity, prompt-retention risk) plus a record-volume uplift; risk = likelihood × impact ÷ 10. Most real LLM data leakage is not exotic model extraction — it is retrieval returning documents the user was never entitled to see, because embeddings were indexed without permission metadata. Prompt logs are the second common route: a general-purpose log store quietly accumulates every secret anyone pasted into a chat window. Memorisation matters mainly for small, repeated fine-tuning sets. The score is a prioritisation estimate for design review, not assurance.
A retrieval layer that ignores the user's permissions turns a chatbot into a search engine for everything the service account can read, which is usually everything.
This calculator takes 8 inputs: Sensitive data in training or fine-tuning, Retrieval scope relative to the user, Multi-tenant isolation, Prompt and output retention, PII redaction before the model sees data, Fine-tune repetition of rare records, Records reachable through the system, Output filtering for sensitive content. The pre-filled defaults are a realistic starting point — replace them with figures from your own environment for a result you can act on.
For confidentiality, generally yes, because you keep the data outside the weights and can filter per request. But it moves the whole burden onto access control at query time — get that wrong and RAG leaks far more readily than a fine-tune would.
Because retention does not make disclosure more likely — it enlarges what is disclosed when something else fails. A breach of an indefinite prompt log exposes years of pasted secrets, not one conversation.