Data and model threats: poisoning, evasion and model theft

Threat 1: data poisoning

An attacker inserts, alters or relabels records in the training or fine-tuning data so that the model learns wrong or hidden behaviour. A common form plants a backdoor: a trigger pattern that makes the model misbehave on demand while it behaves normally on everything else. The asset hit is the training data; the objective broken is integrity; it concerns both predictive and generative systems.

The consequences are wrong decisions at scale, hidden backdoors in production, degraded accuracy and loss of trust in results, plus the cost of retraining from clean data. Retraining cycles reopen the window, which is why the life cycle in lesson 3 marks re-evaluation as a poisoning stage.

Detection relies on provenance and integrity checks on every data source, statistical outlier analysis of training sets, behaviour tests with candidate trigger inputs, and watching for accuracy drift after each retraining. The standard is candid that automated detection remains an open research problem: separating malicious records from harmless outliers is unsolved. Prevention therefore carries most of the weight.

• Mitigate: control who and what can write to training pipelines

• Mitigate: verify and contract data suppliers; keep data snapshots for forensics

• Mitigate: sanitise and validate data before use

• Mitigate: monitor post-deployment performance and roll back on anomalies

Threat 6: direct model poisoning

Here the model itself is altered. Parameters or code are changed in the development or deployment environment, or a supplied pre-trained model arrives with a backdoor already in it. The standard notes a property that makes this the supply-chain version of threat 1: a maliciously trained model keeps its behaviour after the deployer fine-tunes it.

The consequence is hidden malicious behaviour in production that passes normal testing, and dependence on an untrusted supplier for a critical function. Detection uses integrity verification, hashes and signatures on model artefacts, behaviour testing against trigger patterns, and supplier assurance with provenance records.

• Mitigate: secure development environment with access control and change control

• Mitigate: signed, versioned model artefacts from trusted sources only

• Mitigate: an AI bill of materials covering models, data sets and tools

• Mitigate: treat supplied models as supply-chain components under ISO/IEC 27002 supplier controls

Threat 2: evasion

Inputs are crafted so that they look normal to a person but push the model to a wrong output. A slightly altered image passes a content filter; a transaction is shaped to evade fraud scoring; a malware sample is modified to pass a classifier. The asset hit is the model at inference time; the objective is integrity; it mainly concerns predictive systems.

The consequence is that controls which rely on the model silently fail. Fraud passes, prohibited content passes, safety checks pass, and the failure is often discovered only after the loss. Detection uses out-of-distribution and anomaly detection on inputs, flagging of low-confidence or unstable outputs, comparison across an ensemble of models, and adversarial test cases in the validation stage.

Mitigation includes adversarial training, input validation and preprocessing, human review for high-impact low-confidence cases, and not relying on a single model for a critical decision. The standard adds a caution that recurs in clause 7: adversarial training can increase exposure to other attacks. A documented case shows training on adversarial data improved resistance to evasion but increased vulnerability to patch attacks. Controls must be tested together.

Threat 5: model exfiltration

Also called model stealing. The attacker builds a functionally equivalent copy by querying the model many times and training a surrogate on the answers, or obtains the parameters through weak access control. The asset is the model; the objective is confidentiality.

The direct consequence is loss of intellectual property and competitive advantage. The second-order consequence matters more for security: the copy lets the attacker develop evasion and inversion attacks offline, against a model that behaves like the real one, before launching them.

Detection looks for high-volume, systematic query patterns from one client, API usage anomalies against a baseline, and watermark checks on suspected copies. Mitigation combines authentication and per-client rate limiting, output perturbation and limited detail, watermarking of models and outputs, and monitoring backed by contractual terms on API use.

The four threats compared

The table summarises the four threats in this lesson in the standard’s own structure.

Lesson 6 quiz

Five questions. Score 4 or more to mark the lesson complete. Each answer shows an explanation, so a wrong answer teaches as much as a right one.