LLM Data Leakage Risk Calculator

Score sensitive-information disclosure risk in LLM systems: training data, retrieval scope, tenant isolation, logging and memorisation.

Inputs

Sensitive data in training or fine-tuning

Retrieval scope relative to the user

Multi-tenant isolation

Prompt and output retention

PII redaction before the model sees data

Fine-tune repetition of rare records

Records reachable through the systemrecords

Output filtering for sensitive content

Data Leakage Risk

Medium

Risk Score

4.03/ 10

Likelihood of Disclosure

5.75/ 10

Impact of Disclosure

7.00/ 10

Cross-Tenant Leakage Risk

6.00/ 10

Highest-Value Control

Separate indexes per tenant rather than relying on metadata filters

Step by step

Values used
Sensitive data in training or fine-tuning = 6; Retrieval scope relative to the user = 7; Multi-tenant isolation = Shared index with metadata filtering — 5; Prompt and output retention = 6; PII redaction before the model sees data = Partial or pattern-based redaction — 6; Fine-tune repetition of rare records = 5; Records reachable through the system = 250,000 records; Output filtering for sensitive content = Regex only — 5
LLM Data Leakage Risk
likelihood = mean(retrieval scope, redaction gap, memorisation risk, output filtering gap); impact = mean(training-data sensitivity, prompt-retention risk) plus a record-volume uplift; risk = likelihood × impact ÷ 10.
Data Leakage Risk
= Medium
Risk Score
= 4.03 / 10
Likelihood of Disclosure
= 5.75 / 10
Impact of Disclosure
= 7.00 / 10
Cross-Tenant Leakage Risk
= 6.00 / 10
Highest-Value Control
= Separate indexes per tenant rather than relying on metadata filters

How it works

Most real LLM data leakage is not exotic model extraction — it is retrieval returning documents the user was never entitled to see, because embeddings were indexed without permission metadata. Prompt logs are the second common route: a general-purpose log store quietly accumulates every secret anyone pasted into a chat window. Memorisation matters mainly for small, repeated fine-tuning sets. The score is a prioritisation estimate for design review, not assurance. A retrieval layer that ignores the user's permissions turns a chatbot into a search engine for everything the service account can read, which is usually everything.

Formula

LLM Data Leakage Risk

retrievalScope: Whether retrieval respects the requesting user's permissions
memorisationRisk: Chance rare training records are reproduced verbatim
promptLogging: Where prompts and outputs end up and for how long

Frequently Asked Questions

How is LLM Data Leakage Risk calculated?

likelihood = mean(retrieval scope, redaction gap, memorisation risk, output filtering gap); impact = mean(training-data sensitivity, prompt-retention risk) plus a record-volume uplift; risk = likelihood × impact ÷ 10. Most real LLM data leakage is not exotic model extraction — it is retrieval returning documents the user was never entitled to see, because embeddings were indexed without permission metadata. Prompt logs are the second common route: a general-purpose log store quietly accumulates every secret anyone pasted into a chat window. Memorisation matters mainly for small, repeated fine-tuning sets. The score is a prioritisation estimate for design review, not assurance.

Why does LLM Data Leakage Risk matter?

A retrieval layer that ignores the user's permissions turns a chatbot into a search engine for everything the service account can read, which is usually everything.

What values do I need to enter?

This calculator takes 8 inputs: Sensitive data in training or fine-tuning, Retrieval scope relative to the user, Multi-tenant isolation, Prompt and output retention, PII redaction before the model sees data, Fine-tune repetition of rare records, Records reachable through the system, Output filtering for sensitive content. The pre-filled defaults are a realistic starting point — replace them with figures from your own environment for a result you can act on.

Is retrieval-augmented generation safer than fine-tuning?

For confidentiality, generally yes, because you keep the data outside the weights and can filter per request. But it moves the whole burden onto access control at query time — get that wrong and RAG leaks far more readily than a fine-tune would.

Why is prompt retention weighted as impact rather than likelihood?

Because retention does not make disclosure more likely — it enlarges what is disclosed when something else fails. A breach of an indefinite prompt log exposes years of pasted secrets, not one conversation.

You might also need

LLM Prompt Injection Risk Calculator AI Model Security Risk Calculator PII Exposure Calculator