Skip to content
ENExplore Calculators

LLM Data Leakage Risk Calculator

Score sensitive-information disclosure risk in LLM systems: training data, retrieval scope, tenant isolation, logging and memorisation.

About LLM Data Leakage Risk Calculator

Most real LLM data leakage is not exotic model extraction — it is retrieval returning documents the user was never entitled to see, because embeddings were indexed without permission metadata. Prompt logs are the second common route: a general-purpose log store quietly accumulates every secret anyone pasted into a chat window. Memorisation matters mainly for small, repeated fine-tuning sets. The score is a prioritisation estimate for design review, not assurance. A retrieval layer that ignores the user's permissions turns a chatbot into a search engine for everything the service account can read, which is usually everything.

Formula

LLM Data Leakage Risk

likelihood = mean(retrieval scope, redaction gap, memorisation risk, output filtering gap); impact = mean(training-data sensitivity, prompt-retention risk) plus a record-volume uplift; risk = likelihood × impact ÷ 10.

retrievalScope
Whether retrieval respects the requesting user's permissions
memorisationRisk
Chance rare training records are reproduced verbatim
promptLogging
Where prompts and outputs end up and for how long

Worked example

With the default values below, LLM Data Leakage Risk Calculator returns Data Leakage Risk of Medium. Change any input above and every figure here is recalculated in your browser.

Inputs used

Sensitive data in training or fine-tuningone of None — base model only — 1, Internal documents, de-identified — 4, Internal documents with personal data — 7, Customer records or credentials — 10
Internal documents with personal data — 7
Retrieval scope relative to the userone of Filtered to the user's own permissions at query time — 2, Filtered by team — 5, Whole-corpus retrieval regardless of user — 9
Filtered by team — 5
Multi-tenant isolationone of Separate indexes and keys per tenant — 2, Shared index with metadata filtering — 5, Shared index, no filtering — 10, Single tenant only — 1
Shared index with metadata filtering — 5
Prompt and output retentionone of Not retained beyond the session — 2, Retained encrypted with short expiry — 4, Retained indefinitely in a general log store — 7, Sent to a third party for training — 10
Retained indefinitely in a general log store — 7
PII redaction before the model sees dataone of Automated redaction, validated — 2, Partial or pattern-based redaction — 6, None — 9
Partial or pattern-based redaction — 6
Fine-tune repetition of rare recordsone of No fine-tuning — 1, Fine-tuned with deduplication — 4, Fine-tuned on small, repeated data — 8
Fine-tuned with deduplication — 4
Records reachable through the systemaccepts 0–1,000,000,000 records
250,000 records
Output filtering for sensitive contentone of Classifier plus regex on output — 2, Regex only — 5, None — 9
Regex only — 5

Results

Data Leakage Risk
Medium
Risk Score
4.00 / 10
Likelihood of Disclosure
5.00 / 10
Impact of Disclosure
8.00 / 10
Cross-Tenant Leakage Risk
5.00 / 10
Highest-Value Control
Separate indexes per tenant rather than relying on metadata filters

Step by step

  1. 1Values used
  2. 2LLM Data Leakage Risk
  3. 3Data Leakage Risk= Medium
  4. 4Risk Score= 4.00 / 10
  5. 5Likelihood of Disclosure= 5.00 / 10
  6. 6Impact of Disclosure= 8.00 / 10
  7. 7Cross-Tenant Leakage Risk= 5.00 / 10
  8. 8Highest-Value Control= Separate indexes per tenant rather than relying on metadata filters

Assumptions and limits

  • The result is derived only from the 8 inputs listed above. Anything not among them is not part of the calculation.
  • In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
  • Figures are rounded for display to 2 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.

How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.

Frequently Asked Questions

How is LLM Data Leakage Risk calculated?

likelihood = mean(retrieval scope, redaction gap, memorisation risk, output filtering gap); impact = mean(training-data sensitivity, prompt-retention risk) plus a record-volume uplift; risk = likelihood × impact ÷ 10. Most real LLM data leakage is not exotic model extraction — it is retrieval returning documents the user was never entitled to see, because embeddings were indexed without permission metadata. Prompt logs are the second common route: a general-purpose log store quietly accumulates every secret anyone pasted into a chat window. Memorisation matters mainly for small, repeated fine-tuning sets. The score is a prioritisation estimate for design review, not assurance.

Why does LLM Data Leakage Risk matter?

A retrieval layer that ignores the user's permissions turns a chatbot into a search engine for everything the service account can read, which is usually everything.

What values do I need to enter?

This calculator takes 8 inputs: Sensitive data in training or fine-tuning, Retrieval scope relative to the user, Multi-tenant isolation, Prompt and output retention, PII redaction before the model sees data, Fine-tune repetition of rare records, Records reachable through the system, Output filtering for sensitive content. The pre-filled defaults are a realistic starting point — replace them with figures from your own environment for a result you can act on.

Is retrieval-augmented generation safer than fine-tuning?

For confidentiality, generally yes, because you keep the data outside the weights and can filter per request. But it moves the whole burden onto access control at query time — get that wrong and RAG leaks far more readily than a fine-tune would.

Why is prompt retention weighted as impact rather than likelihood?

Because retention does not make disclosure more likely — it enlarges what is disclosed when something else fails. A breach of an indefinite prompt log exposes years of pasted secrets, not one conversation.

Related Calculators

LLM Data Leakage Risk Calculator | Calcrivo