Is membership inference a data breach?

It can be. Confirming that a named person is in a medical, HR or financial training set reveals sensitive information even though no record is copied, and may be reportable under data protection law.

Why does ISO/IEC 27090 stress logging for these threats?

Because the attacker’s queries look like normal use. Without per-client usage logs an organization cannot detect systematic probing or reconstruct what was exposed after the fact.

How does a generative assistant leak documents?

If retrieval sources are indexed without carrying their access permissions, the assistant will answer a user with content from documents that user is not allowed to read. The standard treats retrieval data as needing the same protection as training data.

Lesson 7 of 10 · 13 min read

Information exposure threats: inference, inversion and sensitive output

Last updated: 18 October 2026

Membership inference lets an attacker determine whether a specific record was in a model’s training set. Model inversion reconstructs training data from model outputs. Sensitive model output occurs when a model reveals memorised personal data, secrets or confidential documents it was given at run time. All three are confidentiality threats that leave almost no trace, so ISO/IEC 27090 stresses usage logging, limited confidence scores, differential privacy and access control on retrieval sources.

In this lesson you will learn to

01
Explain membership inference and why it counts as a privacy breach
02
Explain model inversion and what it can reconstruct
03
Describe sensitive model output, including leakage through retrieval
04
Identify the shared mitigations for information exposure threats

Threat 3: membership inference

An attacker queries the model to determine whether a specific record, for example a named person, was part of its training set. Overfitted models answer differently for data they have seen, and carefully constructed inputs expose that difference. The asset compromised is the training data, reached through the model; the objective is confidentiality.

The consequence is a privacy breach without any data theft. Confirming that a person appears in a medical, HR or financial training set can itself be sensitive and reportable under data protection law. Detection is hard because the queries look like normal use. The standard therefore stresses logging model usage so incidents can be reconstructed, analysing query patterns for systematic probing, and running periodic membership inference tests against the organization’s own models.

Mitigate: differential privacy during training, with a documented privacy budget
Mitigate: limit or round the confidence scores returned in outputs
Mitigate: regularisation to reduce overfitting; data minimisation
Mitigate: rate limits and authentication on inference endpoints

Threat 4: model inversion

An attacker uses model outputs, often the confidence scores, to reconstruct training data. Documented cases recover recognisable faces from a face recognition model and sensitive attributes from a scoring model. The asset and objective are the same as membership inference: training data confidentiality, through the model.

The consequence is exposure of personal or proprietary data that was never directly accessible, with regulatory and reputational damage comparable to a data breach. Detection relies on usage logs showing unusual, systematic query sequences, per-client query volume and pattern anomalies, and periodic inversion tests in validation. Mitigation restricts probability and confidence detail in outputs, applies differential privacy and data minimisation, enforces strong authentication with per-client rate limiting, and limits model access to the audiences that need it.

The standard groups membership inference and model inversion as information exposure threats that leave almost no trace in ordinary monitoring. Logging of model usage is the control that makes them investigable at all.

Threat 7: sensitive model output

The model reveals sensitive information in its answers: memorised personal data or secrets from the training data, or confidential content from documents it was given at run time through retrieval. The asset hit is the output; the objective is confidentiality; the threat mainly concerns generative systems.

This is a data breach through the front door. A user with no access to a document can read its content through the assistant that was given it. Detection scans outputs for personal data, secrets and classification markers, red-teams the system with prompts that attempt extraction, and logs outputs for review.

Mitigation starts with minimising and classifying the data used in training and retrieval. The standard makes a specific point that retrieval data in production needs the same protection as training data: enforce the same access control on retrieval sources as on the documents themselves, filter outputs with data loss prevention, and limit memorisation through training choices.

Annex B: direct theft and data leakage

Three Annex B threats belong with this group. Direct model theft is the copying of model files or parameters from storage, repositories or deployment environments. Leakage of training data is unauthorised access to the stores and pipelines that hold training and test sets. Leakage of input data is the exposure of user prompts in logs, caches, vendor systems or shared sessions.

None of these needs a new control. Access control, encryption at rest, repository hygiene, secrets management, classification, separation of environments, log protection, retention limits, vendor contractual terms and tenant isolation are all existing ISO/IEC 27002 controls. The annex records the AI assets they must now cover.

Mitigations shared across exposure threats

Four controls recur across the exposure threats and are worth implementing once, for every externally reachable model.

Control	What it does	Cost or side effect
Usage logging per client	Makes probing visible and incidents reconstructable	Logs are themselves sensitive and need protection and retention limits
Limited or rounded confidence scores	Starves inversion and exfiltration of the signal they need	Less informative output for legitimate users
Differential privacy in training	Bounds what any single record contributes to the model	Accuracy loss; the privacy budget must be managed
Authentication and rate limiting	Caps the volume of queries an attacker can run	Friction for heavy legitimate users

Key terms

Membership inference: An attack that determines whether a specific record was part of a model’s training data.
Model inversion: An attack that reconstructs training data, such as images or attributes, from a model’s outputs.
Differential privacy: A mathematical technique that limits how much any single training record can influence a model, with a measurable privacy budget.
Retrieval-augmented generation: A design in which a generative model is given documents at run time to answer from; those documents can leak through its answers.
Data loss prevention: Controls that detect and block sensitive content leaving a system, applied here to model outputs.

In one paragraph

Membership inference, model inversion and sensitive model output are confidentiality threats that expose training data or run-time documents through a model’s answers, and they leave almost no trace. ISO/IEC 27090 stresses usage logging, limited confidence scores, differential privacy, data minimisation, authentication and rate limiting, and the same access control on retrieval sources as on the documents themselves. Annex B adds direct model theft and data leakage, handled by existing ISO/IEC 27002 controls.

Lesson 7 quiz

Five questions. Score 4 or more to mark the lesson complete. Each answer shows an explanation, so a wrong answer teaches as much as a right one.

Q1What does a membership inference attack reveal?

Q2Which output detail should be limited to hinder model inversion?

Q3Why are membership inference and model inversion hard to detect?

Q4What does ISO/IEC 27090 say about retrieval data used in production?

Q5Which of these is an Annex B threat rather than a clause 6 threat?

Questions people ask

Is membership inference a data breach?: It can be. Confirming that a named person is in a medical, HR or financial training set reveals sensitive information even though no record is copied, and may be reportable under data protection law.
Why does ISO/IEC 27090 stress logging for these threats?: Because the attacker’s queries look like normal use. Without per-client usage logs an organization cannot detect systematic probing or reconstruct what was exposed after the fact.
How does a generative assistant leak documents?: If retrieval sources are indexed without carrying their access permissions, the assistant will answer a user with content from documents that user is not allowed to read. The standard treats retrieval data as needing the same protection as training data.

Shahzad Asghar

Senior IT Strategy and Portfolio Officer at UNRWA, with more than twenty years building and securing data and AI systems across UN agencies. He holds the CISM and CISSP security certifications and writes on AI governance, AI security and assurance, and digital transformation. Connect on LinkedIn.

This course is an independent educational resource by Shahzad Asghar. It is not affiliated with, endorsed by or published by ISO or IEC. ISO/IEC 27090:2026 is copyrighted and is sold through the ISO Store and national standards bodies; this course summarises it and does not reproduce its text. Last reviewed: 18 October 2026.