Foundations of Synthetic Media Generation

GANs, diffusion models, voice cloning, and video generation — and why the engineering details decide which defences can possibly work.

A video call that passed every check a human can make

In January 2024 a finance employee in the Hong Kong office of Arup, the British engineering firm, received a message that appeared to come from the company's UK-based chief financial officer, describing a confidential transaction. The employee was suspicious, and said so. He thought it was phishing.

What resolved his suspicion was a video call. On it were people who looked and sounded like colleagues he recognised, including the CFO. Reassured, he made fifteen transfers totalling HK$200 million, roughly US$25.6 million, to five local bank accounts. Hong Kong police later stated that every other participant on that call was synthetic, assembled from video and audio of the impersonated staff that was already publicly available. Arup confirmed in May 2024 that fake voices and images had been used. C1.01 Established

The detail worth sitting with is not the loss. It is that the employee did exactly what most organisations would call good practice. He doubted a written instruction and sought confirmation through a richer channel. The control he reached for — recognising a senior colleague's face and voice — was the one the attackers had already defeated. His scepticism was routed directly into the trap.

This chapter explains how such artefacts are built. Not as a technical appreciation, but because the generation mechanism determines what traces remain, what a detector can see, how cheaply an attack scales, and whether provenance can help at all. A reader who understands why a diffusion model leaves different residue from a GAN will understand why detector benchmarks published in 2022 stopped predicting field performance in 2024.

Learning objectives

By the end of this chapter you should be able to:

1. Explain the adversarial training loop of a generative adversarial network and why it produces characteristic, detectable artefacts.
2. Describe the forward and reverse processes of a denoising diffusion model, and account for its displacement of GANs in image and video synthesis.
3. Distinguish speaker-adapted from zero-shot voice cloning, and state the data required by each.
4. Compare the computational and data costs of each generation family, and relate those costs to the plausible population of attackers.
5. Assess why detection accuracy measured on one generator rarely transfers to another, using the mechanisms rather than the benchmark numbers.
6. Map a given synthetic artefact to the layer of the trust framework at which it must be addressed.

Two ways to manufacture an image

Nearly all synthetic imagery in circulation comes from one of two architectures. They solve the same problem — produce a sample that could plausibly have come from the training distribution — by almost opposite routes, and the difference matters operationally.

Generative adversarial networks: forgery by competition

The adversarial framework introduced by Goodfellow and colleagues in 2014 sets two networks against each other.C1.02 A generator maps random noise to a candidate image. A discriminator receives a mixture of real and generated images and judges which is which. The generator is rewarded for fooling the discriminator; the discriminator is rewarded for catching it. Training proceeds until, in principle, the discriminator can do no better than chance.

That last point is the chapter's first operational consequence. A GAN's output is shaped by the specific discriminator that supervised it, and the generator's upsampling operations leave periodic structure in the frequency domain that the eye does not register. Early detectors exploited precisely these traces, which is why they performed so well on the generators they were trained against, and so poorly elsewhere.C1.03 Emerging

Architectural work through 2018 and 2019, particularly the style-based generator of Karras and colleagues, pushed facial synthesis to the point where still images became difficult for untrained observers to identify.C1.04 GANs remain fast at inference — a single forward pass — which keeps them relevant for real-time face swapping, the technique most relevant to live video calls.

Diffusion models: forgery by subtraction

A diffusion model works backwards. Training takes a real image and destroys it in small steps, adding Gaussian noise until nothing but noise remains. The network learns to reverse one step: given a noisier image, predict the slightly less noisy one. Generation starts from pure noise and applies that learned step repeatedly.C1.05

This is a more stable training objective than the adversarial game, which is prone to collapse, and it produces greater diversity. The cost is inference: where a GAN generates in one pass, a diffusion model historically required tens to hundreds. Performing the diffusion in a compressed latent space rather than at full pixel resolution cut that cost enough to make the approach practical at consumer scale, which is the change that put image generation in general circulation.C1.06

Voice: the cheapest high-impact modality

Voice deserves separate treatment because its economics differ sharply from video. Two approaches dominate. Speaker adaptation fine-tunes a trained synthesiser on a target's recordings, historically needing substantial clean audio. Zero-shot cloning treats synthesis as a conditional sequence-prediction problem and requires only a short reference sample at inference, with no per-target training at all.C1.07 Emerging

The operational distinction is not audio quality. It is that the second approach removes the per-target cost. Once cloning requires seconds of reference audio and no training run, the constraint on attacker scale stops being compute and becomes target selection. Any executive who has given a conference talk, recorded a welcome video, or appeared on a podcast has published sufficient reference material.

Video: cost collapsing, consistency still hard

Video synthesis splits into two problems that are often conflated. Face replacement on existing footage is mature, computationally modest, and the technique used in fraud: the body, setting, and lighting are real, and only the identity is substituted. Full text-to-video generation creates the scene outright and remains comparatively expensive and prone to temporal inconsistency — objects that drift between frames, physics that does not hold.

Conflating the two leads to a predictable misjudgement of risk. The attacks observed in practice overwhelmingly use the cheap, mature technique against a process weakness, not the expensive, impressive one against perception.

Comparing the generation families

The following comparison is deliberately organised by what each property implies for a defender, rather than by architecture.

Table 1.1 — Generation families by defensive implication. Cost and data columns are relative and change quickly; the defensive column has been more stable. Source: author's synthesis of the cited literature.

Case study: the Arup video-call fraud

Timeline. A written instruction arrives, purportedly from the UK CFO, describing a confidential transaction. The recipient suspects phishing. A video call is convened. The recipient recognises the faces and voices present, and his suspicion is resolved. Fifteen transfers follow. The deception is identified subsequently, after the funds have moved.

Where the controls failed. Not at detection: no detector was deployed, and it is not clear one would have been consulted. The failure sits in three places. First, the payment process permitted a single employee to execute a large, unusual, confidential transfer without a verification path independent of the requester. Second, the organisation's accepted method for resolving doubt — escalate to a richer channel and confirm with a senior person — pointed straight at the compromised medium. Third, and least discussed, the apparent seniority of the participants discouraged further challenge.

What the evidence does not establish. The specific tooling has not been published. Nor is it known whether the employee would have escalated further had the call not occurred, so the counterfactual is unavailable. Reported figures derive from police statements and company confirmation rather than from an independently published forensic report, and the book treats them accordingly.

Transferable lesson. The attack succeeded by anticipating the verification behaviour and pre-positioning inside it. Any control that can be predicted can be targeted, which argues for verification paths the requester cannot influence: a callback to an independently held number, out-of-band confirmation through a second named approver, or a mandatory hold on confidential high-value transfers regardless of who requests them.

Practical scenario

Executive implications

For a board or an executive committee, three consequences follow from the mechanics rather than from the headlines.

• Treat voice and face as identifiers, not authenticators. They establish who someone claims to be, not who they are. Any process that still authenticates on either should be inventoried now and changed on a schedule.
• Do not fund detection as a primary control. Chapters 4 to 7 examine the evidence in detail, but the mechanism in this chapter already predicts the result: detectors trained against one generator family transfer poorly to another, and the attacker chooses the generator.
• Measure the verification path, not the awareness training. The Arup employee was sufficiently aware to be suspicious. Awareness without an uncompromised channel to escalate into converts suspicion to false reassurance.

Common misconceptions

Limitations and open questions

What the evidence in this chapter cannot establish:

• Incidence. There is no reliable denominator. Reported cases are those detected and disclosed, and organisations have strong incentives against disclosure. Any claim about how common these attacks are should be treated as unknown rather than low.
• Attribution of capability. Inferring an attacker's resources from output quality is unsound, because quality now depends more on the tool selected than on the skill applied.
• Durability of traces. Whether the artefacts current detectors rely on will persist across future architectures is an open research question, and the honest answer is that each generation of detectors has so far been overtaken.

Teaching materials

Discussion questions

1. The Arup employee was suspicious and sought verification. Did he behave correctly? If so, what in the organisation failed?
2. Why does the choice between GAN and diffusion architecture change what a defender can measure?
3. Zero-shot voice cloning removed the per-target cost of an attack. Which existing controls does that invalidate in your own organisation?
4. Under what conditions, if any, is perceptual judgement an acceptable control?
5. If incidence is genuinely unknown, how should an organisation size its investment?

Assessment questions

1. Explain the adversarial training objective and derive why it produces generator-specific artefacts. (12 marks)
2. Compare the data and compute requirements of speaker-adapted and zero-shot voice cloning, and state the operational consequence of the difference. (10 marks)
3. A vendor reports 97 per cent detection accuracy. List the questions you would ask before relying on that figure, and justify each. (18 marks)

Practical exercise

Inventory every process in one department that treats a face, a voice, or a video call as an identity check. For each, record the decision it authorises, the value at risk, and whether an independent verification path exists. Report the proportion with no independent path.

Research assignment

Select one published detection benchmark. Establish which generators, codecs, languages and demographic groups it covers, then state precisely which operational claims it can and cannot support.

Seminar topic

"Detection is a tax on defenders and an inconvenience to attackers." Argue for or against, using the generation economics in this chapter.

Further reading

1. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial networks. arXiv. arXiv:1406.2661
2. Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. arXiv. arXiv:2006.11239
3. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. arXiv. arXiv:2112.10752
4. Karras, T., Laine, S., & Aila, T. (2019). A style-based generator architecture for generative adversarial networks. arXiv. arXiv:1812.04948
5. National Institute of Standards and Technology. (2024). Reducing risks posed by synthetic content: An overview of technical approaches to digital content transparency (NIST AI 100-4). nvlpubs.nist.gov
6. Coalition for Content Provenance and Authenticity. (2026). C2PA technical specification, version 2.4. spec.c2pa.org

Chapter synthesis

The generation techniques in this chapter are not equally dangerous, and they are not dangerous in proportion to how impressive they look. The expensive, headline-grabbing capability — generating a convincing scene from a text prompt — has so far featured mainly in disinformation, where volume matters more than precision. The attacks that have moved the largest sums used the cheapest and oldest technique available, face replacement over real footage, aimed at a process that accepted recognition as proof of identity.

That asymmetry is the chapter's contribution to the book's argument. If the determining factor were artefact quality, defence would be a detection problem and would improve as detectors improve. The Arup case points the other way: the artefact had to be good enough to survive a few minutes of a video call among people predisposed to accept it, which is a far lower bar than surviving forensic analysis. Attackers do not need to defeat the state of the art. They need to defeat the weakest check standing between them and a decision.

Mapping this chapter onto Figure 0.1: generation technology sets the cost of an attack at Layer 1, and almost nothing else. It does not determine whether the attack succeeds. Chapters 4 to 7 test whether Layer 2, detection, can carry the weight that most organisations currently place on it. On the evidence of the mechanisms described here, it cannot, and the remainder of the book is largely an argument about what should carry that weight instead.

A Taxonomy of AI-Generated Disinformation Campaigns

With election case studies, and an argument that the decisive variable is not the artefact but the window in which it cannot be answered.

Two days before a national election

On 28 September 2023, two days before Slovakia’s parliamentary election, an audio recording appeared on Facebook. In it, Michal Šimečka, leader of the liberal Progressive Slovakia party, appeared to discuss rigging the vote with Monika Tódová, a journalist at the newspaper Denník N. Fact-checkers established that the audio was synthetic. C2.01 Established

The forgery itself was not especially good. What made it effective was arithmetic. Slovakia observes a 48-hour pre-election moratorium during which media outlets and politicians are expected to stay silent. The recording was published inside that window. The people best placed to rebut it, namely the two individuals impersonated, their party and the newspaper, were the people most constrained from doing so. A further gap compounded it: the clip exploited a limitation in Meta’s manipulated-media policy at the time, which addressed faked video of people saying things they had not said and did not cleanly cover audio. C2.02 Established

This is the observation that organises the chapter. Analysts habitually classify synthetic disinformation by modality, meaning audio, image, video or text, or by how convincing it is. Neither predicts much. The Slovak clip would not have survived a week of scrutiny. It did not need to. It needed to survive 48 hours.

Learning objectives

1. Classify AI-generated influence operations by the condition each requires to succeed, rather than by modality.
2. Define the rebuttal window and explain why it governs impact more reliably than artefact quality.
3. Distinguish attribution, evidence, saturation and process attacks, and identify which defences apply to each.
4. Explain the liar’s dividend and why it inverts the usual relationship between detection and trust.
5. Evaluate the enforcement record of existing legal remedies against documented election cases.
6. Apply the taxonomy to an unfamiliar incident and derive the appropriate institutional response.

A taxonomy organised by requirement

A useful classification tells a defender what to do. Organising by modality does not, because the same defence rarely follows from the same medium. Organising by what the attack requires in order to work does, because each requirement can be denied.

1. Attribution attacks

The attacker places words or actions with a real, identifiable person. Success requires the audience to accept the attribution, and typically requires that the person cannot promptly and credibly deny it. Slovakia is the clean example. So is the New Hampshire robocall discussed below.

What denies it: pre-established, rapid channels of authoritative denial, and provenance on genuine material so that the absence of provenance becomes informative.

2. Evidence attacks

Here the target is not a person but the status of a record. Two directions exist, and the second is the more corrosive. The attacker may fabricate evidence of an event that did not occur, or, more cheaply, may rely on the mere existence of synthetic media to dismiss genuine evidence as fabricated. That second move is known as the liar’s dividend.

3. Saturation attacks

Volume rather than quality. Generative tooling lowers the marginal cost of a plausible artefact to near zero, which allows an attacker to exhaust finite verification capacity. The target is not belief in any single item but the throughput of fact-checkers, moderators and journalists. A defender who measures success by the proportion of items debunked will report progress while losing, because the denominator is set by the attacker.

4. Process attacks

The least discussed and the most directly measurable. The content is instrumental: it exists to change a logistical behaviour, such as whether, when or where someone votes. Success does not require sustained belief, only a decision taken before correction arrives.

Table 2.1 — Taxonomy by requirement. The final column is the chapter’s central claim: for two of the five types, improving detection does not help, and for one of them it actively hurts. Source: author’s taxonomy.

Case study: the New Hampshire primary robocall

Why it is the most instructive election case available. Unlike most incidents, the actor is identified, the method is documented, the actor admitted it, and the legal process ran to a conclusion. It is as close to a controlled experiment in remedy as this field offers.

Classification. It is simultaneously an attribution attack and a process attack. The attribution supplied the authority; the instruction, stay home, was the actual payload. Note that the process element required no sustained belief. A recipient who later doubted the call had already, in some cases, taken the only decision that mattered.

Where the controls failed, and where they did not. Rebuttal was fast: the voice was identified as synthetic quickly and reported widely. Detection was not the gap. The gap was that the telephone network delivered a spoofed-origin political message at scale to a target population, and that nothing in the chain required the caller to be who they claimed.

What the outcome establishes about remedy. This is the finding worth carrying into Chapter 11. The penalties were substantial and the prosecution serious, yet reporting indicates that neither the $6m fine nor a subsequent $22,500 award to voters had been paid. C2.03 Deterrence that arrives nine months after the vote, against a defendant who does not pay, is a weak instrument. Institutions should plan on the assumption that legal remedy will not arrive inside the window that matters.

Evidence limitations. The number of recipients who changed their behaviour is not established, and probably cannot be. Attributing any electoral effect to the calls would exceed the evidence; the case is instructive about mechanism and remedy, not about outcome.

Practical scenario

Executive implications

• Pre-authorise the denial. The binding constraint in a live incident is usually approval, not evidence. Decide now who may issue an authoritative denial without further sign-off, and through which channel, because that decision cannot be taken well at the time.
• Publish provenance on your own output. You cannot stop a forgery being made, but you can make the absence of provenance informative for material attributed to your organisation. This is the only durable answer to the liar’s dividend.
• Do not budget on legal deterrence. The best-documented election case produced a $6m fine that reporting indicates went unpaid, nine months after the vote.
• Triage by consequence. Against saturation, counting debunked items measures the attacker’s output, not your defence.

Common misconceptions

Limitations and open questions

• Effect sizes are unestablished. No documented election case supports a quantified claim about votes changed. Analyses that assert electoral impact generally infer it from reach.
• The taxonomy is analytical, not empirical. It is offered as a tool for deciding what to do, and has not been validated against a labelled corpus of incidents. That validation is a legitimate research project and would be the right way to test it.
• Moratorium interaction is under-studied. Slovakia suggests that pre-election silence rules may be exploitable by design. Whether this generalises across jurisdictions with similar rules is unknown and would be worth comparative work.

Teaching materials

Discussion questions

1. Pre-election moratoria exist to protect deliberation. Does Slovakia show that they now create an exploitable window, and if so what should replace them?
2. Why can detection not address the liar’s dividend, and what can?
3. The New Hampshire fine appears unpaid. What follows for an institution planning its own response?
4. Is labelling synthetic content more useful against fabrication or against denial? Defend your answer.
5. Which of the five attack types is your own organisation least prepared for, and why that one?

Assessment questions

1. Define the rebuttal window and compute it for both case studies, stating your assumptions. (10 marks)
2. Classify three current incidents using Table 2.1 and justify each placement. (15 marks)
3. Construct the argument that improved detection can increase harm, and identify its limits. (15 marks)

Practical exercise

Measure your organisation’s actual rebuttal window. From a hypothetical 02:00 publication of a fabricated statement attributed to your chief executive, how many hours pass until an authoritative denial is live, counting approvals and out-of-hours delay? Most organisations have never measured this and are surprised by the answer.

Research assignment

Select one national election between 2023 and 2026 with a documented synthetic-media incident. Establish the rebuttal window, the platform policy gap if any, and the enforcement outcome, distinguishing evidence from inference throughout.

Seminar topic

“Pre-election moratoria should be abolished because they are now an attack surface.” Argue either side.

Further reading

1. Chesney, R., & Citron, D. (2019). Deep fakes: A looming challenge for privacy, democracy, and national security. California Law Review, 107. Origin of the liar’s dividend argument.
2. International Press Institute. (2023). Slovakia: Deepfake audio of Denník N journalist offers worrying example of AI abuse. ipi.media
3. Federal Communications Commission. (2024). Forfeiture order in the matter of Steve Kramer. See contemporaneous reporting at CyberScoop.
4. AI Incident Database. Incident records for the Slovak audio and the New Hampshire robocall. incidentdatabase.ai
5. National Institute of Standards and Technology. (2024). NIST AI 100-4: Reducing risks posed by synthetic content. nvlpubs.nist.gov

Chapter synthesis

Classifying synthetic disinformation by medium produces tidy categories and poor decisions. Both documented cases in this chapter were audio, both were identified as synthetic quickly, and they failed and succeeded for entirely different reasons. The Slovak clip worked because the response was constrained by electoral rule. The New Hampshire calls were answered within hours, and the remedy still arrived nine months late and, on the reporting, uncollected.

The taxonomy offered here is organised around what each attack needs, because that is what a defender can deny. Its most consequential column is the last one in Table 2.1. For attribution and fabrication attacks, detection contributes something if it is consulted in time. For saturation it is a triage aid at best. For process attacks it is close to irrelevant. And for the liar’s dividend it is counterproductive, because a public trained to expect forgery is a public before whom authentic evidence can be waved away.

Measured against Figure 0.1, this chapter does something uncomfortable to the framework. Chapter 1 argued that Layer 2, detection, cannot carry the weight placed on it. Chapter 2 argues that for a substantial class of attacks Layer 2 is not merely insufficient but the wrong place to look: the operative constraints sit at Layer 4, institutional process, in the form of who may speak and how quickly, and at Layer 5, governance, in the form of whether remedy exists at all. The framework survives this chapter, but only in the version where the layers are not a sequence of filters. They are independent failure points, and the attacker chooses.

Deepfakes as a Cybersecurity Threat Vector

Fraud, identity theft and social engineering. The documented record contains no case in which detecting the fake is what saved the money.

The question that broke a deepfake

In July 2024 an executive at Ferrari received WhatsApp messages apparently from the chief executive, Benedetto Vigna, about an urgent confidential acquisition, followed by a call in a convincing imitation of his voice. The executive did not analyse the audio. He said, in substance, that he needed to identify the caller, and asked which book Vigna had recommended to him a few days earlier. The call ended abruptly. C3.01 Established

That question is the most instructive artefact in this chapter, because of what it did not require. It required no detector, no forensic tool, no judgement about whether the voice sounded right, and no training in synthetic media. It required one fact that both parties held and the attacker could not obtain, asked by the party under pressure rather than offered by the party applying it.

Read the whole documented record of corporate deepfake fraud and the same shape appears. The synthetic artefact never performs the fraud. It supplies authority: it makes an instruction appear to come from someone entitled to give it. The money moves, or does not move, further down the chain, at the point where an organisation decides whether an instruction is authorised. That is the attack surface, and it is a process, not a perception.

Learning objectives

1. Locate the point in an authorisation path at which synthetic media actually operates, and distinguish it from the point at which loss occurs.
2. Explain why no documented corporate case was resolved by detecting the forgery, and what resolved each one instead.
3. Design an out-of-band verification challenge, and identify the three ways such challenges are commonly built wrong.
4. Map where synthetic media enters an identity-verification pipeline, using the FinCEN red-flag set as a reference.
5. Distinguish impersonation fraud from synthetic-identity fraud, which has no victim to call.
6. Assess an existing payment or access-approval control for single-channel dependency.

The authorisation path

An instruction that moves money or grants access passes through five stages. It is worth setting them out plainly, because the habit of discussing deepfakes as a media problem obscures the fact that only one stage is about media at all.

1. Identity claim. Someone asserts who they are. This is the only stage synthetic media touches.
2. Channel. The claim arrives somewhere: a phone line, a messaging app, a video call, an email thread.
3. Request. An instruction is made, usually with urgency and confidentiality attached.
4. Authorisation. Someone or something decides the instruction is legitimate.
5. Execution. The payment leaves, or the credential is handed over.

A forged identity claim is only consequential if stages 2 to 4 permit the same channel that carried the claim to also satisfy the authorisation. Where they do, a good enough voice is sufficient. Where they do not, the quality of the voice is irrelevant. This is why the cases that failed did not fail at stage 1.

The documented record

The table below is restricted to incidents with a named organisation and a primary or first-party source. It excludes the large volume of vendor-reported figures on deepfake fraud growth, which are discussed under limitations.

Table 3.1 — Documented corporate cases with a named organisation. The final column is uniform, and that uniformity is the chapter’s finding rather than an artefact of selection: no case in this set turned on identifying the media as synthetic. Sources: see claims C3.01 to C3.05 and C1.01.

Case study: Retool, and why the deepfake was the least important part

Why it belongs in a cybersecurity chapter rather than a media one. The synthetic voice bought the attacker one additional code from an employee who was already uneasy. On its own that should have been survivable. It was not survivable because a design decision elsewhere had quietly removed a factor: one-time codes synchronised to a cloud account are no longer a second factor once that account is compromised, and the organisation had not registered that change.

The general lesson. Synthetic media does not create new failures so much as collect on failures already present. An authorisation path with a single effective factor was vulnerable to an ordinary pretext call; the voice clone merely raised the success rate. Treating this incident as a deepfake story leads to detector procurement. Treating it as an authorisation story leads to the control that would actually have held.

Identity theft and synthetic identity

Impersonation fraud has a real person behind the claim, which means there is someone to call. Synthetic-identity fraud does not. Here generative tooling is used to manufacture a customer who never existed, through forged identity documents, a face that passes a selfie comparison, and in some cases a live response that passes a liveness check. There is no victim to telephone, and the fraud is only visible as a pattern across accounts.

The United States Financial Crimes Enforcement Network published an alert on this on 13 November 2024, after an increase in suspicious activity reports describing deepfake media in fraud against financial institutions and their customers. It sets out nine red-flag indicators and a reporting key term. C3.06 Established It is a useful reference precisely because it is a supervisory document rather than a vendor paper: the indicators describe inconsistencies between an identity document and the device or geography presenting it, rather than claims about detector accuracy.

Practical scenario

Executive implications

• Audit for single-channel authorisation. For every payment and access-approval path, ask whether the channel that carries the request can also satisfy the approval. Wherever it can, the control is one convincing voice away from failing.
• Make confidentiality a red flag, not an override. Every case in Table 3.1 attached secrecy to urgency. A policy that allows an embargo to suspend verification has inverted its own purpose.
• Check what your second factor still is. Retool is the warning: a factor can be removed by a product change elsewhere without anyone recording the loss.
• Do not train staff to spot deepfakes. Train them to verify out of band regardless. The record shows the second skill works and gives no evidence that the first one does.
• Treat synthetic identity separately. It has no victim to contact, so it is detected across accounts rather than within a call, which makes it a monitoring problem rather than a verification one.

Common misconceptions

Limitations and open questions

• The sample is small and biased toward disclosure. Six named cases are what the public record supports. Organisations that lost money and did not say so are absent, and it is reasonable to assume they outnumber those that spoke. The uniform final column of Table 3.1 should be read as a strong absence of evidence for detection-led saves, not as proof that none has ever occurred.
• Prevalence figures are unreliable. Widely circulated growth rates for deepfake fraud originate largely with vendors selling detection or verification, are rarely accompanied by a published method, and are excluded here. No percentage increase is asserted in this chapter.
• The 2019 case is thinly sourced. It is the most-cited early example and rests on an insurer’s account of a claim, with the company unnamed and the use of AI described as an inference by analysts rather than an established finding. It is retained because it is historically important and is marked accordingly.
• Liveness-attack evidence is weaker than the vendor literature implies. Figure 3.2 describes where attacks enter; it does not quantify success rates against specific commercial systems, and the published independent evidence for such rates is thin.

Teaching materials

Discussion questions

1. If no documented case was decided by detection, what should an organisation buy first, and why is that not the common answer?
2. Ferrari’s executive asked about a book. Construct a challenge that would work in your own organisation, then explain who could defeat it.
3. Retool lost a factor without noticing. What else in your stack could be silently downgraded by someone else’s product decision?
4. Why does synthetic-identity fraud need a different control model from impersonation fraud?
5. Should confidentiality requests be allowed to suspend verification under any circumstances? If yes, specify them precisely.

Assessment questions

1. Using Figure 3.1, explain why artefact quality is a weak predictor of loss. (10 marks)
2. Design an out-of-band verification standard for payments above a threshold, and identify its three most likely failure modes. (20 marks)
3. Compare the control implications of impersonation fraud and synthetic-identity fraud. (15 marks)

Practical exercise

Take one real payment-approval path in your organisation and mark on it the five stages of Figure 3.1. Identify every point at which the same channel could carry both the request and the approval. Count them. In most organisations the number is not zero, and nobody has previously counted it.

Research assignment

Locate a corporate deepfake fraud case with a first-party account published by the affected organisation. Establish what stopped it or failed to, and whether any party made a judgement about the authenticity of the media. Report the evidence quality, not only the narrative.

Seminar topic

“Deepfake awareness training is worse than useless, because it teaches staff to trust their own judgement about artefacts.” Argue either side.

Further reading

1. Financial Crimes Enforcement Network. (2024). Alert FIN-2024-Alert004: Fraud schemes using deepfake media, 13 November. fincen.gov
2. Retool. (2023). Account of the August 2023 social-engineering incident. See also BleepingComputer for the authenticator-sync analysis.
3. MIT Sloan Management Review. (2024). How Ferrari hit the brakes on a deepfake CEO. sloanreview.mit.edu
4. Sophos News. (2019). Account of the UK energy firm voice-fraud case, including its insurer provenance. news.sophos.com
5. AI Incident Database. Records 567 (Retool), 983 (WPP) and 1100 (LastPass). incidentdatabase.ai

Chapter synthesis

The uniform column in Table 3.1 is the chapter in one line. Across six documented corporate cases, not one outcome turned on anybody deciding that a voice or a face was synthetic. The three that failed were stopped by a question about a book, by a messaging app being outside policy, and by a request that did not fit. The three that succeeded succeeded because a single channel was permitted to both make and authorise an instruction.

This reframes the problem in a way that should be uncomfortable for the market that has grown around it. Synthetic media is an attack on stage 1 of a five-stage path. Losses are decided at stage 4. Investment aimed at stage 1 competes with a generator that improves continuously; investment at stage 4 competes with nothing, because the authorisation rule either holds or it does not, and it does not degrade when the models improve.

Against Figure 0.1, the argument from Chapters 1 and 2 now hardens. Chapter 1 showed that Layer 2, detection, cannot carry the weight placed on it. Chapter 2 showed that for timing attacks the binding constraints sit at Layers 4 and 5. Chapter 3 shows something narrower and more actionable: in the fraud case, Layer 4 is not merely where the constraint sits but where the entire outcome is determined, and the controls that work there are old, cheap and already understood. The organisation that cannot answer “can the channel that made this request also approve it?” has a synthetic media problem no detector will solve. That question is the subject of Chapter 12.

Deep Learning and Multimodal Approaches to Deepfake Detection

What detectors actually learn, why benchmark scores do not survive contact with unfamiliar content, and the arithmetic that decides whether a good detector is useful to you.

The model that won did not win

The Deepfake Detection Challenge, run on a purpose-built dataset published by Dolhansky and colleagues in 2020, is the most useful experiment the detection field has produced, and not for the reason it was designed. Entrants were scored on a public validation set and then on a private black-box set of organic videos they had never seen. The best public score was an average precision of 82.56 per cent. The best black-box score was 65.18 per cent, achieved by a model that had ranked fourth on the public set. C4.02 Established

Read that twice. The public leaderboard did not merely overstate performance. It put the models in the wrong order. A practitioner choosing a detector on published benchmark numbers would have chosen a worse one, and would have had no way to know.

This chapter is a technical account of how these systems work, written to explain that result rather than to explain it away. Detection is a real research field with real progress in it, and the previous three chapters have argued that institutions over-rely on it. Both things are true, and the reason they are both true becomes clear once you look at what the signal a detector exploits is actually a signal of.

Learning objectives

1. Describe the main families of deepfake detector and identify the signal each one exploits.
2. Explain why in-distribution accuracy and cross-generator accuracy diverge, in terms of what the model has fitted.
3. Order detector families along the distance between their signal and a specific generator, and state the trade-off this creates.
4. Compute the precision of a detector at a realistic prevalence, and explain why published accuracy figures are insufficient for a deployment decision.
5. Specify what to require from a detection vendor instead of a benchmark score.
6. Identify the conditions under which detection is genuinely the right control.

What a detector is fitted to

Nearly all published visual detectors are binary classifiers over cropped faces or short face-tracked clips. The architecture varies; the learning problem usually does not. A model is shown authentic and manipulated examples and learns whatever feature best separates them in that collection. The question that decides operational value is therefore not how accurate the model is, but what the separating feature turns out to be.

The reference training collection for most of the field is FaceForensics++, published by Rössler and colleagues at ICCV 2019: 1,000 authentic videos from YouTube and 4,000 manipulated versions produced by four methods, two of which change identity, by face swapping, and two of which change expression, by reenactment. C4.01 Established It is a good dataset and it was an enormous contribution. It is also four synthesis pipelines, and a model trained on it can reach high accuracy by learning the characteristic residue of those four pipelines rather than anything about forgery as such.

That distinction is the whole chapter. A feature that identifies this generator is cheap to learn, scores extremely well on a test split drawn from the same generators, and has no reason whatever to survive a generator it has not seen. A feature that identifies a property of the synthesis process, or an inconsistency that any synthesis would have to produce, is harder to learn, scores lower in-distribution, and transfers better. The field’s history is largely a migration from the first kind of feature to the second.

Close to the generator

The earliest effective detectors were ordinary image classifiers, often XceptionNet, applied to aligned face crops, and frequency-domain methods that examined the spectral signature left by the upsampling operations in a generator’s decoder. Both work well and both are, in effect, generator fingerprinting. Retrain the generator, change its upsampling, compress the video differently, and the fingerprint moves.

Away from the generator

Three later families deliberately avoid generator-specific residue. Blending-boundary methods target the composite seam that any face-swap pipeline has to produce when it inserts one region into another image, which is a property of the process rather than of a particular model. Temporal-semantic methods target motion that real speech constrains: LipForensics, published by Haliassos and colleagues at CVPR 2021, pretrains a spatio-temporal network on lipreading, then fine-tunes on fixed mouth embeddings, explicitly so that the classifier attaches to high-level mouth dynamics rather than to low-level manipulation artefacts. C4.04 Established Cross-modal methods go furthest, asking whether the audio and the visible articulation agree, which no single-modality generator is obliged to get right.

The direction of travel is sound and the reasoning behind it is correct. It is also, read honestly, an admission: the field moved away from its best-scoring features because those features were measuring the wrong thing.

The transfer gap, in numbers

Table 4.1 — Only published figures from the primary sources are reproduced, with the comparison each paper itself makes. No current state-of-the-art number is given; see limitations for why. Sources: claims C4.01 to C4.04.

The arithmetic nobody runs before purchase

Suppose you obtain a detector that is genuinely good on your content: 90 per cent sensitivity and 90 per cent specificity, which is better than the DFDC black-box result. Now deploy it where synthetic items are rare, which is the normal case for an institutional content pipeline. The quantity you care about is not accuracy; it is precision, the share of flagged items that are actually synthetic.

Case study: the Deepfake Detection Challenge as a natural experiment

Why no later benchmark has replaced it as evidence. Subsequent datasets are larger and harder, and published scores on them are higher. But very few experiments since have held back a genuinely unseen set and published the rank correlation between the two conditions. That second element is the contribution. Any benchmark can tell you a model is accurate on itself; this one told the field that its selection procedure was broken.

What it does not establish. It is one challenge, on one dataset, at one moment in the development of synthesis. The specific figures should not be quoted as current detector performance, and nothing here implies that detection has failed to improve since 2020. The durable finding is structural, not numerical: when the test distribution is drawn from the training generators, the leaderboard measures familiarity rather than capability.

Practical scenario

Executive implications

• Never accept a benchmark score as a procurement input. Ask for results on generators held back from training, the rank order across both conditions, and the operating threshold with its calibration. A vendor who reports only AUC is reporting the one number DFDC showed to be misleading.
• Compute your own prevalence first. Figure 4.2 is arithmetic and takes five minutes. It frequently ends the procurement.
• Price the false positive explicitly. In this domain it is not a nuisance; it is self-inflicted reputational damage to authentic material.
• Use detection as triage, never as adjudication. A score routes an item to a human. It does not settle anything, and it must never be quoted externally as a finding.
• Expect to retrain, and budget for it. A detector is a perishable control whose shelf life is set by somebody else’s release schedule.

Common misconceptions

Limitations and open questions

• No current state of the art is asserted. Leaderboards move faster than a book can, and the durable finding is the structure of the transfer gap rather than any peak figure. Quoting a 2026 number here would date the chapter and would also repeat the error it describes.
• Benchmarks remain narrow. The standard datasets are face-centric, mostly derived from a small number of synthesis pipelines, and weighted toward video of talking people. Detection of synthetic documents, satellite imagery, scientific figures and screen recordings is comparatively unstudied.
• Operational degradation is under-reported. Published evaluations use curated clips. Real submissions are recompressed, cropped, re-encoded, screen-recorded and passed through platforms that alter them. Robustness to that chain is rarely the headline measurement.
• Calibration is almost never published. Without it, a threshold chosen on one distribution means something different on another, which makes the false-positive cost in Figure 4.2 unknowable in advance for most commercial systems.
• Evasion is the subject of Chapter 7. Everything in this chapter assumes an attacker who is not specifically targeting your detector. That assumption is generous and is withdrawn later.

Teaching materials

Discussion questions

1. The DFDC black-box winner ranked fourth publicly. What does that imply about every published detector comparison that reports one test condition?
2. Why does moving the exploited signal away from the generator reduce peak accuracy? Is that reduction avoidable in principle?
3. At what prevalence would you accept a detector’s output as a reason to withhold publication? Defend the number.
4. Cross-modal detection needs a synchronised speaking face. What fraction of the content your organisation handles meets that condition?
5. If a detector is a perishable control, what does that imply for how it should be procured and budgeted?

Assessment questions

1. Derive precision from sensitivity, specificity and prevalence, and plot it for one detector at four prevalences. (10 marks)
2. Explain the transfer gap in terms of what the classifier has fitted, using FaceForensics++ as the example. (15 marks)
3. Write the detection section of a procurement specification. State what must be reported and why each item is necessary. (20 marks)

Practical exercise

Estimate the prevalence of synthetic items in one real content pipeline in your organisation. Most teams cannot do this, and discovering that you cannot is the finding: it means any detector you deploy there has an unknown precision.

Research assignment

Find a published detector evaluation that reports results on generators held back from training and gives the rank order across conditions. Report how many papers you had to read to find one.

Seminar topic

“Detection benchmarks have harmed the field by optimising for a metric that does not transfer.” Argue either side.

Further reading

1. Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Nießner, M. (2019). FaceForensics++: Learning to detect manipulated facial images. ICCV. arXiv:1901.08971
2. Dolhansky, B., et al. (2020). The DeepFake Detection Challenge (DFDC) dataset. arXiv:2006.07397
3. Li, Y., Yang, X., Sun, P., Qi, H., & Lyu, S. (2020). Celeb-DF: A large-scale challenging dataset for DeepFake forensics. CVPR. openaccess.thecvf.com
4. Haliassos, A., Vougioukas, K., Petridis, S., & Pantic, M. (2021). Lips don’t lie: A generalisable and robust approach to face forgery detection. CVPR. arXiv:2012.07657
5. Chollet, F. (2017). Xception: Deep learning with depthwise separable convolutions. CVPR. The backbone used as the FaceForensics++ detection baseline.

Chapter synthesis

Detection is a legitimate research field that has made real progress, and this chapter has tried to represent it accurately rather than convenient. Its central technical story is a migration: from features that identify a particular generator, which score superbly and transfer badly, toward features grounded in properties any synthesis must exhibit, which score less well and transfer better. That migration is the correct response to the problem, and it has a ceiling, because the further a signal sits from the generator the less of it there is to measure.

Two independent failures stand between a published score and an operational control. The first is distributional: DFDC showed not only that performance drops on unfamiliar material but that the ranking inverts, so the benchmark cannot even be trusted to choose between models. The second is arithmetical and has nothing to do with machine learning at all: at institutional prevalences, precision collapses, and the errors it produces are false accusations against authentic material, which is the specific harm Chapter 2 identified as the one detection cannot fix.

Against Figure 0.1 this settles the status of Layer 2. Detection is not a filter the other layers sit behind; it is an instrument with a narrow operating envelope, defined by whether you can observe the generator set, whether prevalence is high enough, and whether the output feeds a process rather than a verdict. Inside that envelope it is valuable, and Chapter 5 examines the audio case where the envelope is tighter still. Outside it, deploying detection does not reduce risk. It relocates it.

Voice Cloning and Audio Deepfake Detection

Three seconds is enough to clone a voice. The channel that carries nearly every real attack is also the channel that destroys the evidence a detector needs.

Three seconds

In January 2023 Microsoft researchers published VALL-E, which treats speech synthesis as language modelling over discrete audio codec tokens. Pretrained on 60,000 hours of English speech, it synthesises a target speaker from a three-second enrolled recording of a speaker it has never heard, and preserves the emotion and acoustic environment of that prompt. C5.01 Established

This resolves the claim Chapter 1 left open. The cost of cloning a voice is no longer a training run against a corpus of your target; it is three seconds of audio. Every executive who has given a conference talk, every official who has answered a reporter, every manager who has recorded a voicemail greeting has already published the enrolment prompt.

That fact alone would make this a short chapter: voice can no longer function as an identity claim, and Chapter 3 showed that identity claims were never where losses were decided anyway. But audio detection deserves a fuller account than that, because it is where the detection field has its longest and most honest evidence base, and where that evidence points to something more specific and more awkward than in the video case.

Learning objectives

1. State the current cost of cloning a voice and the implication for voice as an authentication factor.
2. Explain how narrowband and lossy coding remove the signal that audio countermeasures rely on.
3. Interpret ASVspoof results correctly, distinguishing closed-condition challenge performance from field performance.
4. Describe what the in-the-wild evaluation established and the limits of its headline figure.
5. Decide where in an audio pipeline detection should be applied, if at all.
6. Specify the controls that hold for voice attacks without requiring any judgement about the audio.

Where the evidence lives, and where it goes

An authentic recording and a synthetic one differ in ways that are mostly small and mostly high. Vocoder and codec-decoder output tends to leave traces in the upper spectrum, in the fine structure of harmonics and noise, and in the statistical regularity of phase. A countermeasure trained on full-band studio audio has a great deal of that material to work with.

Now send the same audio through a telephone network. Classical narrowband telephony encodes roughly 300 to 3,400 Hz at an 8 kHz sampling rate under ITU-T G.711. C5.04 Established Everything above 4 kHz is simply absent. Wideband voice over IP does better, up to around 7 kHz, but applies perceptual coding designed to discard anything a listener will not miss, which is a reasonable description of the artefacts a detector is hunting. Add packet loss concealment, automatic gain control and re-encoding by a messaging platform, and the residue that distinguished the two recordings has been attenuated by processes that are entirely legitimate and completely routine.

What the challenge record establishes

Audio has a better evidence base than video, because the ASVspoof series has run repeated community challenges on shared data since 2015, with published overviews. ASVspoof 5, reported in 2024, is the most informative edition: crowdsourced speech from almost two thousand speakers, an order of magnitude more than earlier editions, adversarial attacks included for the first time, and 53 participating teams. Many submissions performed well. The overview also records that performance degrades under adversarial attacks and under the application of neural encoding and compression schemes. C5.03 Established

That second sentence is the one to carry away. It is not a complaint from a critic; it is the challenge organisers reporting that the conditions which make a deployment realistic are the conditions in which the field’s best systems lose ground.

The complementary result comes from outside the challenge setting. Müller and colleagues, at Interspeech 2022, assembled 37.9 hours of found audio of public figures, of which 17.2 hours were deepfakes, and evaluated existing methods on it. Their finding, in the paper’s own words, is that related work performs poorly on such real-world data, with performance degradation of up to one thousand per cent. C5.02 Established

Table 5.1 — The three sources this chapter rests on, with their limits stated beside them. As in Chapter 4, no current equal error rate is asserted. Sources: claims C5.01 to C5.03.

Case study: the attack channel paradox

Why this is more than an inconvenience. It means the deployment most people imagine, an audio detector sitting on the corporate telephone system or screening voice notes, is operating in close to the worst available conditions on close to the least informative copy of the signal. It also means the research setting and the operational setting are separated by a transformation that nobody is doing anything wrong by applying.

What follows for control design. Exactly what Chapter 3 established, and from a second independent direction. The control that holds for a voice attack does not involve the audio: a question whose answer the caller cannot obtain, a callback to a number from your own directory, an approval that cannot be satisfied through the channel that made the request. None of these degrade with bandwidth.

Practical scenario

Executive implications

• Remove voice as a sole or primary authentication factor. Three seconds of your executives’ recorded speech is already public. This is a policy change, not a procurement.
• Do not expect benchmark behaviour on telephone audio. The published conditions and your trunk are separated by a bandwidth reduction that removes most of the detector’s evidence.
• Detect on the best copy, as early as possible. If audio detection is in scope, apply it to the highest-fidelity recording available before transcoding, never to a downstream re-encode.
• Retain original audio where law allows. Most organisations cannot produce the original of a disputed recording, which forecloses both detection and provenance. Chapter 10 takes this further.
• Buy the channel-independent control first. Out-of-band verification works at 3.4 kHz, at 48 kHz, and on a transcript. It is also cheaper.

Common misconceptions

Limitations and open questions

• The thousand per cent figure is relative and bounded. It is the paper’s own characterisation of degradation against the specific baselines it evaluated in 2022. It should not be quoted as a general multiplier, and it is not evidence about any current system.
• No current equal error rate is asserted. As in Chapter 4, the durable finding is structural. Challenge results improve annually and the gap between closed and field conditions has proved more persistent than any single score.
• Where spoofing residue sits is reported, not quantified here. Figure 5.1 marks a region qualitatively. A rigorous version would state, per vocoder family, how much discriminative information survives a given codec, and that measurement is not consistently published.
• Real-time cloning is Chapter 9. This chapter treats synthesis as something produced before the call. Interactive cloning changes the operational picture and is handled with the other emerging threats.
• Adversarial evasion is Chapter 7. ASVspoof 5 includes adversarial attacks for the first time, and the degradation it reports under them belongs to that discussion.

Teaching materials

Discussion questions

1. If a voice is a public identifier rather than a secret, what processes in your organisation currently treat it as a secret?
2. Why does the same compression that makes a call intelligible make it harder to assess? Is there any way to have both?
3. The challenge organisers themselves report degradation under codecs. Why has that not changed procurement behaviour?
4. What would it take to retain original-fidelity audio for disputed calls, and what are the privacy consequences of doing so?
5. Ferrari’s executive asked a question rather than listening harder. Why is that the general solution rather than a lucky one?

Assessment questions

1. Explain, in terms of bandwidth and coding, why audio countermeasures degrade on telephone channels. (10 marks)
2. Given a benchmark equal error rate and a prevalence, compute the precision of a contact-centre deployment and state the consequence for genuine callers. (15 marks)
3. Argue for or against retaining voice biometrics in a customer-authentication flow, given a three-second cloning cost. (20 marks)

Practical exercise

Find three seconds of clean public speech for a named executive in your own organisation, using only public sources, and time how long the search takes. Do not synthesise anything. The finding is the search time, and in most organisations it is under a minute.

Research assignment

Take one published audio countermeasure with available code and evaluate it on the same audio before and after narrowband coding. Report the change. State clearly what your experiment does and does not establish.

Seminar topic

“Voice biometrics should be withdrawn from financial authentication entirely.” Argue either side, costing the customer-experience consequences.

Further reading

1. Wang, C., et al. (2023). Neural codec language models are zero-shot text to speech synthesizers (VALL-E). arXiv:2301.02111
2. Müller, N., et al. (2022). Does audio deepfake detection generalize? Interspeech. isca-archive.org
3. ASVspoof consortium. (2024). ASVspoof 5: Crowdsourced speech data, deepfakes and adversarial attacks at scale. arXiv:2408.08739
4. Yamagishi, J., et al. (2021). ASVspoof 2021: Accelerating progress in spoofed and deepfake speech detection. arXiv:2109.00537
5. ITU-T Recommendation G.711. Pulse code modulation of voice frequencies. The narrowband baseline referred to throughout this chapter.

Chapter synthesis

The audio case sharpens the argument of Part II rather than repeating it. Chapter 4 found two reasons a published detector score does not transfer: the test distribution is drawn from the training generators, and precision collapses at institutional prevalence. Audio adds a third, and it is the most structural of the three. The signal a countermeasure depends on is concentrated where the communication channel is designed to discard information, and the channels that do most of that discarding are precisely the channels through which every documented voice attack has arrived.

This is not a gap that better models close, because nothing is malfunctioning. Narrowband coding is working as specified. Perceptual coding is doing the job it was designed for. An organisation screening telephone audio for synthetic speech is asking a detector to read evidence that the telephone system has already, correctly, thrown away.

Against Figure 0.1, Layer 2’s envelope narrows again, and Layer 1, generation, has moved decisively: three seconds and a public recording. What has not moved is Layer 4. The question Chapter 3 ended on, whether the channel that carried the request can also satisfy the approval, is bandwidth-independent, works on a transcript, and would have stopped every case in Table 3.1. Chapter 6 turns to why the benchmarks that produce these scores are constructed the way they are, and what it would take to build one that predicted field behaviour.

Benchmarking, Datasets, and Cross-Domain Generalization

Detectors expire for reasons that have nothing to do with attackers, and the benchmarks that certify them cannot see it happening.

The paper that found the signal and published its cure

In 2020 Durall, Keuper and Keuper showed that the up-sampling operations inside generative convolutional networks fail to reproduce the spectral distribution of natural images. Transposed convolutions add high-frequency noise; interpolation-based up-sampling leaves high frequencies missing. They noted that these spectral distortions can be used to detect generated data, reporting up to 100 per cent accuracy on public benchmarks. C6.01 Established

In the same paper they proposed a spectral regularisation term for the generator’s training objective, which removes the high-frequency error, stabilises training and improves output quality.

Read those two paragraphs together. One publication identified a detection signal good enough for near-perfect benchmark accuracy, and supplied the modification that eliminates it, offered not as an evasion technique but as an improvement to image quality. Any generator author who adopted it for the stated reason, better pictures, destroyed the detector as a side effect and would have had no idea they had done so.

This is the mechanism this chapter is about. Detection capability decays for reasons entirely unconnected to anyone attacking a detector. Deliberate evasion exists and is the subject of Chapter 7. Obsolescence is different: it is automatic, it is driven by ordinary progress in generation, and it is invisible to the measurement apparatus the field uses to certify progress.

Learning objectives

1. Distinguish detector obsolescence from adversarial evasion, and explain why the first is the larger operational problem.
2. Identify the axis along which detection benchmarks have scaled and the axis that actually governs transfer.
3. Explain why a cross-dataset performance drop does not identify its own cause.
4. State what a transfer-honest evaluation must report, and audit a published paper against that list.
5. Describe the comparability problem that unified benchmarking frameworks were built to address.
6. Decide what evidence would justify trusting a detector on content it has not seen.

The axis the field did not scale

Look at how the reference datasets grew. FaceForensics++, in 2019, contained 1,000 authentic videos and four manipulation methods. Celeb-DF raised synthesis quality. DFDC raised scale, added organic video and introduced a genuine black-box condition. Each was a real contribution and each made the data harder.

Now ask what each one varied. Mostly identities, resolution, compression, lighting, pose, and the sheer number of clips, which is to say the content and the conditions. The number of distinct generator families stayed small throughout. That matters because the evidence says generator family is the variable that governs whether a detector transfers, and content diversity is not a substitute for it.

Corvi and colleagues demonstrated the point directly at ICASSP 2023: detectors built for GAN-generated imagery struggle on images from diffusion models, and, in the stronger result, even when retrained on diffusion output they often fail to generalise across different diffusion models. C6.02 Established A detector can be trained on a vast, varied, high-quality corpus and still be defeated by a change of architecture it has never met.

Why a cross-dataset drop does not explain itself

The standard robustness experiment is to train on dataset A and test on dataset B, then report the fall. It is better than not doing it, and it is much weaker evidence than it looks, because A and B differ simultaneously in generator family, synthesis quality, compression history, resolution, capture conditions, demographics and labelling policy. A drop from 0.95 to 0.62 AUC is a fact about two datasets. It attributes nothing.

This is the methodological gap that unified benchmarking was created to close. DeepfakeBench, presented at NeurIPS 2023 by Yan and colleagues, was motivated by exactly this: data-processing pipelines were not uniform, so detection models were not receiving comparable inputs; experimental settings differed noticeably; and evaluation strategies and metrics were not standardised. It supplies a unified data-management layer, a common implementation framework for 15 detection methods across 9 datasets, and standardised protocols. C6.03 Established

The existence of that contribution is itself a finding worth stating plainly. A field in which published numbers are not comparable to one another had been reporting progress against those numbers for several years.

Table 6.1 — The right-hand column is a procurement specification. Very few published evaluations supply more than two of its rows. Source: author’s synthesis of the issues raised by C4.02, C6.02 and C6.03.

Case study: a detection signal with a published expiry date

What makes this the cleanest available illustration. The incentive to adopt the fix is not evasion. It is picture quality and training stability, which is what generator authors are paid to improve. A detection signal whose removal is a by-product of the generator community’s own objectives is not a durable control, however well it scores.

The general pattern. A signal is discovered, published, and adopted by detectors. Publication tells generator authors precisely which property of their output is anomalous. They remove it, usually for quality reasons. The detector expires, silently, and the benchmark continues to report the old number because the benchmark’s generators have not changed.

Practical scenario

Executive implications

• Treat a detector as a perishable with an unknown expiry date. It does not fail loudly. It degrades while its flag rate looks normal, so plan a re-validation cadence rather than waiting for a signal that will not come.
• Hold back your own test set, and never share it. A private set of material from generators your supplier has not trained on is the only instrument that tells you anything a benchmark cannot.
• Require a per-generator breakdown. An aggregate figure hides the weakest case, which is the one an attacker will choose.
• Ask what would make this detector obsolete. A supplier who cannot answer has not thought about the lifecycle in Figure 6.2.
• Do not interpret a stable flag rate as evidence of health. It is equally consistent with a detector catching only the generators it already knew.

Common misconceptions

Limitations and open questions

• Figure 6.1 is a judgement, not a measurement. There is no accepted metric for generator family diversity, so the positions express an argument about direction of travel. Constructing such a metric would be a genuinely useful contribution and would allow the claim to be tested or refuted.
• Leave-one-generator-out is necessary and not sufficient. Generator families are not cleanly separable: architectures share components, training data overlaps, and many deployed systems are proprietary and cannot be enumerated at all. A held-out protocol improves the evidence without closing the gap.
• No current benchmark results are quoted. As in Chapters 4 and 5, the finding here is about measurement structure. A current leaderboard figure would date immediately and would also be the kind of number this chapter argues against relying on.
• The lifecycle in Figure 6.2 is not quantified. How long a published signal survives in practice is an empirical question, and a study tracking signal lifetimes across generator releases does not appear to exist.
• Deliberate evasion is excluded here by design. Chapter 7 adds an attacker who is specifically targeting the detector, which is a different and faster failure mode.

Teaching materials

Discussion questions

1. Durall and colleagues published both a detection signal and the modification that removes it. Should they have? What follows for publication norms in this field?
2. Why has the community scaled content diversity so much more readily than generator diversity? Consider cost, licensing and credit.
3. If a stable flag rate cannot distinguish a working detector from an expired one, what instrumentation would?
4. Is a benchmark whose numbers are not comparable across papers worse than no benchmark?
5. Which rows of Table 6.1 could your organisation actually require from a supplier, and which would be refused?

Assessment questions

1. Explain why a drop from 0.95 to 0.62 AUC across datasets does not attribute a cause, and design an experiment that would. (15 marks)
2. Distinguish obsolescence from evasion, with a worked example of each. (10 marks)
3. Write a leave-one-generator-out evaluation protocol, and state what it still cannot establish. (20 marks)

Practical exercise

Take any published detection paper and score it against the six rows of Table 6.1. Record the score. Then do the same for the marketing material of a commercial detector, and compare.

Research assignment

Pick one published detection signal and trace what happened to it: when it was reported, which detectors adopted it, and whether subsequent generator releases retained or removed the underlying property. Report how far you could get with public information.

Seminar topic

“Detection research should not publish exploitable signals, for the same reasons vulnerability disclosure is coordinated.” Argue either side.

Further reading

1. Durall, R., Keuper, M., & Keuper, J. (2020). Watch your up-convolution: CNN based generative deep neural networks are failing to reproduce spectral distributions. CVPR. arXiv:2003.01826
2. Corvi, R., et al. (2023). On the detection of synthetic images generated by diffusion models. ICASSP.
3. Yan, Z., Zhang, Y., Yuan, X., Lyu, S., & Wu, B. (2023). DeepfakeBench: A comprehensive benchmark of deepfake detection. NeurIPS Datasets and Benchmarks. github.com/SCLBD/DeepfakeBench
4. Marra, F., et al. (2019). Do GANs leave artificial fingerprints? Early evidence for generator-specific traces.
5. Dolhansky, B., et al. (2020). The DFDC dataset. arXiv:2006.07397 — for the rank-inversion result in Table 6.1.

Chapter synthesis

Part II has now produced four independent reasons a published detector score does not transfer to a deployment, and only one of them is about machine learning being hard. Chapter 4: the test distribution is drawn from the training generators, and the ranking can invert. Chapter 4 again: precision collapses at institutional prevalence, as arithmetic. Chapter 5: the operational channel removes the evidence, by design and correctly. Chapter 6: the signal expires because generator authors improve their products, and the benchmark cannot observe the expiry because its generators are fixed.

The fourth is the one that should change behaviour, because it is the only one that keeps acting after procurement. A detector bought on sound evidence, validated on your own archive and performing visibly well can be substantially obsolete eighteen months later with no outward sign, and the single most common health indicator, a stable flag rate, is consistent with both outcomes. Durall and colleagues showed the whole mechanism inside one paper: find the anomaly, publish it, and supply the fix that quality-conscious engineers will adopt for their own reasons.

This closes the last of the open literature-direction claims in the ledger. Chapter 1 asserted that early detectors exploited generator-specific up-sampling traces and generalised poorly, and marked it Emerging pending this chapter. It is now pinned: Durall for the trace, Corvi for the failure to transfer across generator families, including within diffusion. Against Figure 0.1, Layer 2 can be stated precisely at last. Detection is a measurement of a relationship between a detector and a generator population that someone else controls and continuously changes for reasons of their own. It is useful inside the envelope Chapter 4 specified, it expires on a schedule nobody publishes, and it was never the layer that was going to hold. Chapter 7 removes the last generous assumption by introducing an adversary who is aiming at the detector on purpose.

Adversarial Robustness and Evasion of Deepfake Detectors

Flipping the lowest bit of every pixel turns a 0.95 AUC detector into one that is reliably wrong. And if that happened to you, nothing would tell you.

Below chance

Carlini and Farid, at the CVPR workshops in 2020, took a classifier that reached an AUC of 0.95 across almost all existing image generators while trained on only one, and attacked it five ways. With full access they flipped the least significant bit of each pixel and the AUC fell to 0.0005. Perturbing one per cent of the image area gave 0.08. A single added noise pattern gave 0.17. A black-box attack, without access to the classifier, gave 0.22. C7.01 Established

Note what those numbers mean. An AUC of 0.5 is chance. Four of the five attacks drove the detector below chance, which is a stronger result than breaking it: a detector at 0.0005 is not confused, it is inverted. Its output has become a reliable indicator of the opposite of the truth, and an institution acting on it would do better to flip every answer.

Two of those figures matter more than the headline. The single noise pattern at 0.17 is one perturbation, computed once, reusable without per-image optimisation. The black-box figure of 0.22 required no access to the model at all. Hold that beside the finding of Chapter 6, that detectors do not transfer across generator families, and the asymmetry of this field becomes visible. The attacks generalise. The defences do not.

Learning objectives

1. State the measured effect of white-box and black-box evasion on a strong detector, and explain what an AUC below 0.5 implies operationally.
2. Explain why the transferability of attacks and the non-transferability of detectors is a structural asymmetry rather than a coincidence.
3. Describe why adversarial perturbations survive the compression chain, and cite the evidence.
4. Identify the absent feedback channel in content authenticity, and contrast it with fraud.
5. Explain why “we have observed no evasion” is uninformative in this domain.
6. Decide what an institution should do given an unmeasurable evasion rate.

The objection that does not hold

The standard reply to adversarial results is that they are laboratory artefacts: a perturbation at the level of the least significant bit cannot possibly survive a platform upload, a transcode, a resize and a recompression. It is a reasonable objection and it has been tested.

Hussain, Neekhara and colleagues, at WACV 2021, built white-box and black-box pipelines that fool neural deepfake detectors into classifying fake videos as real, and state in their abstract: “We further demonstrate that our adversarial perturbations are robust to image and video compression codecs, making them a real-world threat.” C7.02 Established

An attacker who wants the perturbation to survive the pipeline can simply include the pipeline in the optimisation. There is nothing exotic about this. It is the same technique that makes adversarial road signs work in sunlight.

The asymmetry, stated plainly

Table 7.1 — Every row favours the attacker, but the fifth is the one without a precedent in ordinary security practice. Sources: C7.01, C7.02, C6.02; final row is analysis, C7.03.

The missing feedback channel

Security controls improve because failures are observable. A fraud control fails and money leaves an account, so somebody reconciles a statement and the organisation learns. A spam filter fails and a user sees the message. An intrusion detection system fails and, eventually, an incident is discovered. The loss signal is what converts an attack into an improvement.

Content authenticity has no such signal. When a detector is successfully evaded, the item is classified as authentic and nothing further happens. It is published, or believed, or filed, exactly as a genuine item would be. There is no reconciliation, no complaint, no breach notification, because from the institution’s side the two outcomes are identical. C7.03 Analysis

The consequence is uncomfortable and worth stating without hedging. The real-world evasion rate of any deployed content detector is unmeasurable in principle by the organisation deploying it. Not merely unmeasured. There is no instrument, because the quantity has no observable consequence. An organisation that has run a detector for three years without incident has exactly the same evidence as an organisation that has been evaded weekly for three years.

Case study: five attacks on one classifier

Why the target choice matters. This was not a weak or outdated model picked to make a point. It was a detector with the cross-generator property the field had been pursuing, which is to say the better the detector, the more there was to break.

What the universal pattern implies for cost. A single reusable noise pattern dropped the AUC to 0.17. That is not an attack requiring expertise per item; it is an attack requiring expertise once. The marginal cost of evading every subsequent item is the cost of adding a pattern to an image.

What it does not establish. One classifier, in 2020, under the authors’ own experimental conditions. Detectors have changed and some have been hardened. The result establishes the mechanism and its cheapness, not a current figure for any particular system.

Practical scenario

Executive implications

• Never report “no known evasions” as assurance. It is guaranteed output regardless of the truth. Reporting it as comfort is the single most common error in this area and it misleads boards in a specific, correctable way.
• Generate your own evidence, since the environment will not. Red-team the detector deliberately: commission adversarial submissions and measure what gets through. This is the only way to obtain the loss signal Figure 7.2 shows the system does not produce.
• Do not publish detector verdicts externally. A published score is an attack surface in both directions, and the inbound direction discredits authentic material.
• Assume evasion is cheap for a motivated actor. A universal pattern at 0.17 AUC and a black-box attack at 0.22 are not state-actor capabilities.
• Keep the control at Layer 4. Every finding in Part II points the same way: out-of-band verification and authorisation design do not have an evasion rate, because there is nothing to perturb.

Common misconceptions

Limitations and open questions

• The figures are from 2020 and from one classifier. They establish mechanism and cost, not current performance of any system. A fair reading is that detectors have improved and so have attacks, and the asymmetry in Table 7.1 is unchanged by either.
• No claim is made about how often evasion occurs in practice. The chapter argues such a claim is unavailable in principle from the defender’s side. Anyone asserting a field evasion rate should be asked how they measured it.
• The both-directions attack is reasoned, not demonstrated here. The cited work perturbs synthetic media toward “authentic”. The reverse direction follows from the mechanism, and is marked as analysis rather than established.
• Certified defences exist for narrow threat models. This chapter does not survey them. The relevant limitation is that certification is relative to a perturbation bound the attacker is under no obligation to respect.
• Nothing here implies detection is worthless. Chapter 4 specified an envelope in which it earns its place. This chapter narrows that envelope where an adversary is deliberately present, which is not every deployment.

Teaching materials

Discussion questions

1. An AUC of 0.0005 is worse than useless. Is an inverted detector more dangerous than no detector, and to whom?
2. Why do attacks transfer across detectors when detectors do not transfer across generators? Is there a reason to expect this asymmetry to persist?
3. Design the cheapest honest experiment that would give your organisation a real evasion signal.
4. Should detector verdicts ever be made public? Consider both attack directions.
5. Adversarial training trades clean accuracy for robustness. Using Figure 4.2, what does that trade cost you at your prevalence?

Assessment questions

1. Explain what an AUC below 0.5 means and why it is qualitatively different from poor performance. (10 marks)
2. Construct the argument that a deployed detector’s evasion rate is unmeasurable, and identify the strongest objection to it. (20 marks)
3. Compare the feedback structure of a fraud control and a content detector, and derive two consequences for governance. (15 marks)

Practical exercise

Write the three-sentence board answer from the scenario above for a control your organisation actually operates. Then identify who currently believes the reassuring version.

Research assignment

Find a published claim about how often deepfake detectors are evaded in the real world. Trace it to its method. Report what you find, including if you find no method.

Seminar topic

“Controls whose failures are unobservable should not be relied upon for consequential decisions, however accurate they appear.” Argue either side, with examples from outside this field.

Further reading

1. Carlini, N., & Farid, H. (2020). Evading deepfake-image detectors with white- and black-box attacks. CVPR Workshops. openaccess.thecvf.com
2. Hussain, S., Neekhara, P., Jere, M., Koushanfar, F., & McAuley, J. (2021). Adversarial deepfakes: Evaluating vulnerability of deepfake detectors to adversarial examples. WACV. cseweb.ucsd.edu
3. Neekhara, P., et al. (2021). Adversarial threats to deepfake detection: A practical perspective. On transferability across detectors.
4. ASVspoof consortium. (2024). ASVspoof 5. arXiv:2408.08739 — for degradation under adversarial attack in the audio case, C5.03.
5. Durall, R., Keuper, M., & Keuper, J. (2020). arXiv:2003.01826 — for the contrast between obsolescence and evasion, C6.01.

Chapter synthesis

Part II is now complete, and it has produced five independent reasons that a detector’s published performance does not constitute an institutional control. The test distribution is drawn from the training generators, and the ranking can invert. Precision collapses at institutional prevalence, as arithmetic. The operational channel removes the evidence, correctly and by design. The signal expires because generator authors improve their products, and the benchmark cannot see it. And an adversary who is actually aiming at the detector can drive it below chance for the cost of one reusable noise pattern, with perturbations that survive the codecs.

The fifth reason is the one that changes governance rather than procurement, and it is the absent feedback channel rather than the attack itself. Every other control an institution runs reports its own failures eventually, because failure has a consequence someone notices. This one does not. An evaded detector says “authentic” and the organisation proceeds, and the comfort that accumulates with years of quiet service is manufactured by the structure of the problem rather than earned by the control.

Against Figure 0.1, Layer 2 can now be closed out. Detection is an instrument with a narrow envelope, an unpublished expiry date, a cheap and transferable evasion path, and no means of telling you when it has failed. Part III turns to the threats that are arriving rather than established, and Part IV to the layers that do not share these properties: provenance, which attaches evidence to authentic material rather than hunting for traces in hostile material, and institutional process, which has held in every documented case in this book so far.

LLM-Generated and Multimodal Disinformation

Text detectors do not measure provenance. They measure fluency, and that is why their errors land on the same people every time.

Sixty-one per cent

Liang and colleagues, publishing in Patterns in 2023, ran seven commercial GPT detectors over TOEFL essays written by non-native English speakers. The average false-positive rate was 61.22 per cent. All seven detectors unanimously flagged 19.8 per cent of those human-written essays as machine-authored, and at least one detector flagged 97.8 per cent of them. For essays by native English writers the false-positive rate was roughly five per cent. C8.02 Established

Then the detail that settles what these tools are actually measuring. The authors reduced the misclassification by using ChatGPT to enrich the vocabulary of the TOEFL essays, which brought the average false-positive rate down from 61.3 to 11.6 per cent.

Putting an LLM through your essay makes it look less like an LLM wrote it. No instrument measuring provenance could behave that way. These detectors estimate how statistically unsurprising a text is, usually through perplexity, and a writer with a narrower range of English expression produces exactly the signature they are built to flag. The false positives are not noise around a working measurement. They are the measurement.

Learning objectives

1. Explain why text offers no generator artefact of the kind images and audio provide, and what detectors substitute for one.
2. State the measured disparity in false-positive rates between native and non-native English writers.
3. Describe the effect of recursive paraphrasing on watermarked and non-watermarked detection.
4. Summarise the theoretical limit on detection as human and machine text distributions converge.
5. Distinguish the generation question from the coordination question, and say which is actionable.
6. Decide where AI-text detection may and may not be used in decisions affecting people.

Why text is different

Part II rested on a physical fact: a generator leaves traces in a sampled signal. Up-sampling disturbs a spectrum, a vocoder leaves residue in harmonics, a composite leaves a seam. Those are properties of a process acting on a continuous medium, and even when they are perishable they are real.

Text has nothing equivalent. It is discrete, short, and carries no sampling process to fingerprint. There is no pixel grid and no spectrum. What remains is the statistical shape of the word sequence, which is to say its style. And style is also the thing that distinguishes one human writer from another, so any detector operating on it necessarily conflates two different notions of unusual: produced by a machine, and written by an atypical human.

Who counts as atypical is not random. It is the second-language writer, the writer using a formulaic register because that is what their training taught, the writer with a smaller active vocabulary, the writer working in a technical template. The measured twelve-fold disparity is the predictable consequence of the design, not a bug awaiting a patch.

Case study: the maker withdraws its own detector

Why this is the strongest available evidence on text detection. It is first-party, it is adverse to the publisher’s commercial interest, and the organisation with the most complete knowledge of how the text was generated reached the conclusion that it could not reliably detect it. Every commercial claim to the contrary should be read against that.

Run Chapter 4’s arithmetic on it. Sensitivity 0.26, specificity 0.91. At a prevalence of one AI-written item in twenty, precision is about 13 per cent. At one in a hundred, about 2.8 per cent. Analysis The tool was not merely weak. Deployed at plausible rates it would have been wrong about roughly nineteen flags in twenty, and each wrong flag is an accusation against a named person.

Paraphrasing, and a limit that is not engineering

Suppose a detector were much better, and suppose the text carried a deliberate statistical watermark. Sadasivan and colleagues tested this with recursive paraphrasing. A watermarking scheme’s true-positive rate at a one per cent false-positive rate fell from 99 per cent to 15 per cent after five rounds of paraphrase, with AUROC falling from 99.8 to 67.9 per cent. Neural detectors, zero-shot classifiers and retrieval-based detectors were also broken, the last being specifically designed to resist paraphrase. C8.03 Established

The same work supplies something more fundamental than an attack. It connects the AUROC of the best possible detector to the total variation distance between the human and machine text distributions, and derives an impossibility result: as those distributions converge, even an optimal detector performs only marginally better than random.

This is a different kind of claim from anything in Part II. The detector problems in Chapters 4 to 7 were contingent: benchmarks were built badly, signals expired, channels removed evidence, adversaries perturbed inputs. Each could in principle be improved. Here the ceiling is set by how similar machine text is to human text, and the entire generative research programme is a sustained effort to make them more similar. Progress in generation is not an attack on text detection; it is the same quantity, measured from the other side.

The question worth asking instead

For disinformation at scale, “was this text generated by a machine” is close to the least useful question available. A hostile influence operation is not characterised by the provenance of its sentences. It is characterised by being an operation: many accounts, one intent, coordinated in time, provisioned from a common source.

Coordination leaves traces that are not in the content at all. Account creation patterns, posting synchrony, shared infrastructure, identical engagement timing, the ordering in which a claim appears across venues. Paraphrasing removes the stylistic signal and leaves every one of those untouched, because they are properties of behaviour rather than of text.

Table 8.1 — The right-hand column is better on five rows and not clean on any of them. Row three is the reason it must not be sold as a solution. Source: author’s analysis, C8.04.

Practical scenario

Executive implications

• Do not use AI-text detection in any decision affecting a person. Not admissions, hiring, performance, discipline, visas, benefits or procurement. A documented twelve-fold disparity against non-native speakers makes this a discrimination mechanism with a technical finish, and the disparity is foreseeable rather than accidental.
• If such a tool is already in use, the exposure is live. The evidence is published, the mechanism is understood, and foreseeability is the pivot on which Chapter 11 turns. Withdraw it before you are asked to justify it.
• Never ask someone to prove they wrote something. It is not provable, the request is a sanction in itself, and it falls hardest on those least able to contest it.
• Redirect disinformation spending from content to coordination. Paraphrase defeats the first and not the second, and only the second produces an enforceable finding.
• For your own text, mark provenance rather than detect its absence. Declaring what you produced is tractable; proving what someone else produced is not. Chapter 10 takes this up.

Common misconceptions

Limitations and open questions

• The bias figures are from seven detectors in 2023. Commercial tools have changed and vendors claim improvement. The mechanism argument, that perplexity-like measures penalise narrower expression, is what carries forward; specific rates should be re-measured rather than assumed.
• The impossibility result is theoretical and carries assumptions. It bounds the best possible detector under its framework. It is not a claim that no text is ever detectable, and short, heavily templated machine output can remain distinguishable.
• Coordination detection is asserted as better-posed, not as validated. No figures are offered for its accuracy, and the box above states its disparate-impact problem. Treating it as the answer would repeat the error this chapter documents.
• Multimodal coordinated content is under-measured. Operations now pair generated text with generated imagery and voice, and there is no established evaluation for detecting the combination as opposed to each stream separately.
• No current commercial accuracy claim is assessed here. As throughout the book, vendor figures without a published method are excluded rather than argued with.

Teaching materials

Discussion questions

1. If enriching an essay’s vocabulary with an LLM reduces AI-detection flags, what exactly is being detected, and what word should we use for it instead of detection?
2. OpenAI withdrew its own classifier. Why has that not ended the commercial market for text detection?
3. Is a tool with a twelve-fold disparate error rate ever acceptable if its aggregate accuracy is high?
4. Coordination analysis has its own false-positive victims. How would you set the threshold, and who should decide it?
5. Does the impossibility result mean institutions should stop asking whether text was generated? What should they ask instead?

Assessment questions

1. Compute precision for the withdrawn OpenAI classifier at two prevalences and interpret the result for a named individual. (10 marks)
2. Explain why text lacks a generator artefact and what detectors substitute, then derive the disparate-impact result from first principles. (20 marks)
3. Contrast the generation question and the coordination question across the rows of Table 8.1, and state where the latter is also unsafe. (15 marks)

Practical exercise

Find out whether any part of your organisation currently runs AI-text detection on material submitted by people. Most organisations contain at least one such tool that no committee ever approved.

Research assignment

Take any commercial AI-text detector and locate its published false-positive rate broken down by writer first language. Report what you find, including finding nothing.

Seminar topic

“Deploying AI-text detection on work submitted by people is indefensible on the published evidence.” Argue either side.

Further reading

1. Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023). GPT detectors are biased against non-native English writers. Patterns, 4(7). cell.com/patterns
2. Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., & Feizi, S. (2023). Can AI-generated text be reliably detected? arXiv:2303.11156
3. OpenAI. (2023). New AI classifier for indicating AI-written text, and its subsequent withdrawal. openai.com
4. Kirchenbauer, J., et al. (2023). A watermark for large language models. ICML. The scheme whose paraphrase vulnerability is reported in C8.03.
5. Carlini, N., & Farid, H. (2020). CVPR Workshops — for the parallel evasion result in the image case, C7.01.

Chapter synthesis

Part II established that detection is a narrow instrument badly understood. Text is worse, and worse in a different way. The problems are no longer contingent. A detector on text cannot measure provenance because there is no trace of provenance to measure, so it measures statistical typicality instead and converts a question about machines into a judgement about how ordinary someone’s writing is. The consequence has been measured: 61.22 per cent false positives for non-native English writers against roughly five per cent for native writers, 97.8 per cent of human-written essays flagged by at least one of seven tools, and a false-positive rate that falls when an LLM is allowed to improve the vocabulary.

Above that sits a ceiling that engineering does not lift. Detection quality is bounded by the distance between the human and machine distributions, and closing that distance is the purpose of the field producing the generators. Chapter 6 showed detectors expiring as a side effect of quality work; here the same dynamic applies to the best detector that could ever exist.

The institutional conclusion is unusually blunt for this book, because the evidence is unusually clear. Text detection should not be used in decisions about people, and organisations currently using it are running a documented disparate-impact mechanism whose harm is foreseeable from published literature. The useful question is not whether a sentence was generated but whether a set of accounts is an operation, which paraphrase cannot defeat and which at least yields an enforceable finding. That question also harms people when answered badly, and saying so is part of answering it honestly.

Against Figure 0.1, Layer 2 has now failed on text for reasons of principle rather than practice, and Layer 3, platform and distribution, is where the coordination question lives. Chapter 9 turns to the threats still arriving: real-time cloning, text-to-video, agentic impersonation and synthetic identities at scale.

Emerging Generative Threats

Real-time cloning, text-to-video, agentic impersonation and synthetic identities. A stress test of this book’s own recommendation against the capabilities arriving next.

An advisory that arrives at the same control

On 15 May 2025 the FBI’s Internet Crime Complaint Center issued public service announcement I-051525-PSA, describing a campaign active since April that sent smishing texts and AI-generated vishing voice messages impersonating senior US federal and state officials. The targets were current and former officials and the people in their address books, and the objective was to move recipients onto other platforms and links in order to harvest credentials and take over accounts. C9.01 Established

The advisory’s own recommended protection is to verify any unsolicited message claiming to come from an official by calling a known number.

That is the control this book derived in Chapter 3 from the fraud record, and again in Chapter 5 from the physics of the telephone channel. A federal law-enforcement body facing an active campaign arrived at it independently. This chapter is where that convergence gets tested, because the capabilities now arriving are aimed at precisely the human judgements the control does not depend on, and it is worth being exact about which is which.

Learning objectives

1. Distinguish controls that rest on content judgement from controls that rest on channel control.
2. Explain why real-time synthesis defeats the first category and not the second.
3. List the human verification heuristics already consumed by synthesis capability, and the ones still intact.
4. Describe how agentic tooling changes the economics of social engineering rather than its mechanism.
5. Identify the single case in which this book’s recommended control has no counterparty, and state what replaces it.
6. Assess an emerging capability for which of your controls it actually consumes.

Content judgement versus channel control

Every verification practice in use falls into one of two families, and the distinction decides what survives.

Content judgement asks somebody to assess the artefact. Does this sound like her? Does the face move correctly? Is the lighting consistent? Did the caller answer my question fluently? Every member of this family has a generative capability pointed at it, and each capability that lands removes a heuristic permanently.

Channel control asks nothing about the artefact. I ended the call and dialled a number from my own directory. The approval requires a second person I contacted separately. The challenge requires a fact that is not published. These do not improve or degrade with synthesis quality, because the artefact is not the evidence. The evidence is the path.

Read the FBI advice again with that distinction in hand. “Calling a known number” is not advice about listening carefully. It is advice about who initiates and to what address, and it would hold if the impersonation were flawless.

What each arriving capability actually consumes

Table 9.1 — Three of four arriving capabilities consume perception and leave channel control standing. The fourth is the exception, and it is the exception because it removes the person rather than imitating them. Source: author’s analysis, C9.03.

Does real-time cloning kill the callback?

This is the question the chapter exists to answer, and the honest answer requires separating two things the callback was doing.

It was never doing what people assume. Its strength is not that a live conversation is hard to fake, and a book that recommended it on those grounds would now have to withdraw the recommendation. Real-time synthesis is operationally proven: the Arup transfers followed a multi-party video call in which several apparent colleagues were synthetic. Interactivity as a perceptual test is finished. C1.01

What the callback actually does is shift the attacker’s task from producing convincing content to controlling a channel you selected. To defeat a callback to a number held in your own directory, an attacker must compromise the number, the directory, or the network, which is a materially different and more expensive operation than cloning a voice from a conference talk. The quality of the clone they can produce once you reach them is irrelevant, because you are no longer using the clone as evidence.

Agentic impersonation: an economic change, not a new mechanism

Autonomous agents that hold a multi-turn conversation, maintain a pretext across days, consult what they have been told and act on a system do not introduce a new attack. They change the price of the attack documented in Chapter 3.

Social engineering has been expensive because it needed a skilled human per target, improvising in real time, with limited patience. That cost is why the documented cases are a handful of high-value attempts against named executives rather than a flood. Remove the per-target human and the attribution attack of Chapter 2 acquires the volume property of the saturation attack, and the authorisation paths of Chapter 3 get probed exhaustively rather than once.

Synthetic identities: the one real gap

Here the recommended control fails, and it is worth saying so plainly rather than stretching it. A callback verifies that the person who contacted you is the person you believe them to be. When the identity is manufactured, there is no genuine person behind it and nothing for a callback to reach. The documents are generated, the face passes comparison, and the whole apparatus of out-of-band verification has no counterparty.

What replaces it is not verification but monitoring, and the shift is from the individual interaction to the population. FinCEN’s indicators are instructive on exactly this point: they are predominantly inconsistencies between artefacts and behaviours, such as device or geographic data that does not match the identity document presented, or a newly opened account with rapid volume to high-risk payees, rather than verdicts about whether a document is real. C3.06

The institutional implication is uncomfortable for the structure of most organisations. Impersonation is caught at the point of interaction, by whoever is being spoken to. Synthetic identity is caught across accounts, over time, by a function that sees the whole population. Those are different teams, usually different systems, and often different budgets.

A capability worth noticing for a different reason

Text-to-video with synchronised dialogue arrived in mainstream products in 2025: Sora 2, released on 30 September, generates video with synchronised dialogue, sound effects and lip-synced speech, and includes a likeness feature in which a person uploads a short reference video so that their face and voice can be used, under a consent framework giving them visibility of videos featuring them and the ability to revoke access. C9.02 Established

The generative capability is the expected news. The consent framework is the more interesting development, because it is the first mainstream instance of a likeness being handled as a permissioned asset with an audit trail and revocation, implemented inside a generation product rather than imposed on it. That is a governance control at the point of synthesis, which is where Chapters 10 and 11 argue the durable answers live. Whether such frameworks are robust, and what happens to likeness rights when they are not, is Chapter 11’s subject.

Practical scenario

Executive implications

• Rewrite “verify out of band” into the three explicit properties. Most policies state the principle and leave the staff member to infer the mechanics, which is where the scenario above goes wrong. You initiate, to an address you already held, on a different channel.
• Stop investing in perceptual training. Four for four, the heuristics consumed so far were content judgements. Training staff to spot synthetic media is training in a skill with a known expiry.
• Protect the address book, not only the principal. The FBI campaign targeted officials’ contacts. Your executives’ counterparties are your attack surface, and they are outside your controls.
• Audit for coverage gaps, not control strength. If agentic tooling makes attempting every path cheap, the forgotten path matters more than the hardened one. Enumerate paths before strengthening any.
• Assign synthetic identity to a population-level function. It cannot be caught in the interaction, so giving it to the people who answer the phone guarantees it is missed.

Common misconceptions

Limitations and open questions

• This is the most speculative chapter in the book, and it is marked as such throughout. Two claims are established from primary sources. The agentic argument is explicitly Emerging, and the heuristic ledger is analysis over claims established elsewhere.
• No prevalence or growth figures appear. Emerging-threat discussion is where vendor statistics are least reliable and most abundant. None are used, and the absence is deliberate.
• Real-time synthesis quality is not characterised. Latency, failure modes under interruption and robustness to unexpected questions are the properties that would decide how well interactive impersonation holds up, and they are not consistently published.
• The consent-framework observation is about design, not efficacy. Whether likeness permissioning is robust in practice, or circumvented by other tooling, is untested here.
• Agentic predictions are falsifiable and unfalsified. The two stated consequences have not been measured. If attack volume against mid-value targets does not rise, the argument is wrong and should be withdrawn.

Teaching materials

Discussion questions

1. Why were all four consumed heuristics content judgements? Is there a reason to expect the next one to be too?
2. The FBI reached this book’s recommendation independently. Why is that stronger evidence than either arriving at it alone?
3. Does a likeness consent framework inside a generation product represent governance, or the appearance of it?
4. Synthetic identity breaks the recommended control. What would it take for your organisation to detect it at population level?
5. If agentic tooling makes attempting every path cheap, is your weakest path or your strongest control the better thing to measure?

Assessment questions

1. Define content judgement and channel control, and classify six verification practices from your own organisation. (15 marks)
2. Explain why real-time synthesis does not defeat a callback, and identify the conditions under which it would. (15 marks)
3. State the agentic-impersonation argument, its evidence status, and the observation that would refute it. (10 marks)

Practical exercise

Find your organisation’s written guidance on verifying unusual requests. Check it against the three properties. Most such guidance says “verify independently” and specifies none of them, which means the control exists on paper and is improvised in practice.

Research assignment

Locate a documented case of autonomous agentic impersonation conducting an end-to-end fraud. Report the evidence quality honestly, including if the strongest available source is a vendor blog post.

Seminar topic

“Deepfake awareness training should be discontinued and the budget moved to authorisation design.” Argue either side, using Figure 9.1.

Further reading

1. Federal Bureau of Investigation, IC3. (2025). Public service announcement I-051525-PSA, on smishing and AI-generated vishing impersonating senior US officials.
2. OpenAI. (2025). Sora 2 is here, 30 September. openai.com/index/sora-2
3. Financial Crimes Enforcement Network. (2024). Alert FIN-2024-Alert004. fincen.gov — for the population-level indicator approach, C3.06.
4. AI Incident Database. Record 1077, on the FBI-reported campaign. incidentdatabase.ai
5. Wang, C., et al. (2023). arXiv:2301.02111 — for the three-second cloning cost underlying row two of Figure 9.1, C5.01.

Chapter synthesis

Part III was supposed to be the chapter where the book’s recommendation came under pressure, and in one respect it did. Interactivity is gone as a perceptual test, and gone with operational proof rather than in principle: the Arup transfers followed a synthetic multi-party video call. Anyone whose verification rests on a live exchange feeling right has lost the control they thought they had.

But the pattern across the four arriving capabilities is consistent, and it is not an accident. Synthesis competes with perception. It has no purchase on who initiated a call, which directory supplied the number, whether the approver was contacted separately, or whether a challenge requires an unpublished fact. Every heuristic consumed so far belonged to the first category and every heuristic still standing belongs to the second, which is why a federal advisory confronting an active campaign recommends calling a known number rather than listening harder.

Two honest qualifications keep this from being a comfortable conclusion. Agentic impersonation is logged as anticipated rather than documented, with two falsifiable predictions attached, and if attack volume against mid-value targets does not rise the argument should be withdrawn. And synthetic identity genuinely breaks the recommendation, because manufacturing a person who never existed leaves nobody to call back. That case moves from verification to population-level monitoring, which in most organisations is a different team with a different budget, and it is the clearest instance in this book of a threat whose control does not sit where the threat is noticed.

Against Figure 0.1, Part III closes with Layers 1 and 2 settled and the weight now entirely on Layers 3 to 5. Part IV takes them in turn: provenance, which attaches evidence to authentic material instead of hunting traces in hostile material; the legal and regulatory position, including whether a remedy ordered is ever a remedy collected; and the organisational framework that the last nine chapters have, one case at a time, been specifying.

Content Provenance and Watermarking

Provenance asks a bounded question about material you control, instead of an unbounded one about material an adversary controls. It becomes useful only at the point where absence starts to mean something.

The inversion

Everything in Parts II and III shared a structure. A hostile artefact arrives, and an institution asks whether it is genuine. The adversary chose the artefact, the generator, the channel and the moment; the institution has an instrument with a narrow envelope, an unpublished expiry date and no feedback when it fails.

Provenance reverses every term. Instead of asking whether an artefact an adversary produced can be shown to be false, it asks whether an artefact you produced can be shown to be yours. You control the capture device, the editing pipeline, the signing key and the publication path. Nothing about the question depends on predicting what a generator will do next.

That is why this chapter opens Part IV, and why the book turns constructive here. But the inversion comes with a condition that is usually left out of provenance advocacy, and it determines whether any of this pays: a credential on authentic material carries information only in proportion to how unusual its absence is. Early in adoption, an unsigned file tells you nothing, because almost everything authentic is also unsigned. The value is not in the signature. It is in the eventual significance of its absence.

Learning objectives

1. Explain how provenance inverts the burden of proof relative to detection, and why that changes the scaling properties.
2. Describe the C2PA manifest structure and distinguish hard bindings from soft bindings.
3. Explain what durable Content Credentials are for and what weakness they inherit.
4. State what provenance establishes and what it cannot, including the authenticated-lie failure mode.
5. Summarise the published attacks on watermarking in both the removal and spoofing directions.
6. Identify the threshold condition that determines when provenance becomes operationally useful.

How Content Credentials are built

The Coalition for Content Provenance and Authenticity specifies a structure worth understanding in outline, because its security properties follow from it. Assertions record things about an asset: metadata, the actions performed on it, thumbnails, cryptographic bindings. The assertions referenced together are collected into an assertion store; combined with additional information they form a claim; the claim is digitally signed, producing a claim signature; and the assertions, claim, signature and supporting information together form the C2PA manifest. C10.01 Established

Two kinds of binding tie a manifest to content, and the difference matters more than any other technical detail in this chapter.

A hard binding is cryptographic. It lets a validator confirm that this manifest belongs to this asset and that the asset has not been modified, by computing values that can match only this asset and no other, not even assets derived from it or other renditions of it. The specification provides byte-range hashing for arbitrary formats, box-based hashing for formats such as JPEG, PNG and GIF, BMFF-based hashing for ISO Base Media File Format files, and collection hashing for multi-asset workflows. A manifest may contain no more than one hard-binding assertion.

A soft binding is statistical. It is a content identifier that is either not statistically unique, such as a fingerprint, or embedded as an invisible watermark, and it is computed from the content of an asset rather than its raw bits, which is what allows it to survive into derived assets and alternative renditions. A manifest may contain zero or more.

C2PA’s version 2 architecture has been put through the ISO fast-track route as ISO/DIS 22144, Authenticity of information — Content credentials.

What the published attacks establish

Watermarking is the component most often presented as the answer, so it deserves the book’s standard treatment. Saberi and colleagues, at ICLR 2024, analysed this rigorously and the results cut in two directions.

For watermarking methods with low perturbation budgets, they demonstrate a fundamental trade-off between the evasion error rate, meaning watermarked images detected as unmarked, and the spoofing error rate, meaning unmarked images detected as marked, when diffusion purification attacks are applied; and they show that diffusion purification removes such watermarks effectively while applying minimal change to the image. For high perturbation budget methods, where visible-scale changes are applied, diffusion purification is ineffective, and the authors instead develop a model substitution adversarial attack that removes the watermark successfully. C10.02 Established

The spoofing direction is the one that receives least attention and deserves most. Removing a mark lets synthetic content pass as unmarked. Adding a mark to content that was never generated by the marked system lets an attacker manufacture a provenance claim, and in the inverse case lets them have authentic material reported as machine-made. That is the Chapter 2 liar’s dividend again, now with a technical artefact to point at.

Table 10.1 — The last row is a governance question disguised as a technical one. Sources: C10.01, C10.02, C10.03.

Verification you cannot perform is not verification

Watermarking at scale is now deployed. Google DeepMind’s SynthID embeds its signal into the content itself rather than into metadata, across image, audio, text and video outputs. The text variant was published in Nature in October 2024 and open-sourced, with a production implementation shipping in a widely used machine-learning library. The image, audio and video watermarks are proprietary, and confirming them requires the vendor’s own detector. C10.03 Established

Embedding into content rather than metadata is the right engineering decision, for the reason Figure 10.1 makes plain: metadata does not survive the pipeline. But it is worth naming what the proprietary case does to the trust relationship. A cryptographic credential can be checked by anyone holding a trust list, which means the claim is independently verifiable and the verifier does not have to believe the issuer. A proprietary watermark can be checked only by the party that made it. The answer is then not a verification but a referral, and an institution relying on it has transferred its judgement to a vendor rather than acquired the means to exercise its own. For a newsroom or a court that distinction is the whole question.

The threshold property

The most common objection to provenance is that credentials are easily stripped, so the scheme is pointless. The most common defence is durable credentials. Both miss what actually determines the value, which is the information content of an absent credential, and that depends entirely on adoption.

This explains a pattern that otherwise looks like institutional inertia. Each organisation evaluating provenance individually finds a weak business case, correctly, because a signature on its own output changes very little while everyone else’s output is unsigned. The returns are in the third panel and they are collective. That is an argument for standards bodies, procurement mandates and sector-wide commitments rather than for persuading firms one at a time, and it is the strongest reason the ISO route matters more than any single product feature.

Case study: durable Content Credentials

Why this is the right design anyway. The alternative is a credential that works only in the minority of cases where metadata survives, which would never reach the third panel of Figure 10.2. Accepting a weaker assurance on the recovery path in exchange for coverage is a reasonable trade, and it is a trade rather than a solution.

What follows for interface design. Because the two recovery paths carry different assurances, an honest implementation distinguishes them. A single badge that appears identically whether the manifest arrived by cryptographic hard binding or by watermark lookup is telling the user less than the system knows.

Practical scenario

Executive implications

• Sign your own output; do not try to detect everyone else’s. This is the one control in the book whose cost and difficulty you fully control, and whose effectiveness does not decay as generators improve.
• Treat the signing key as the asset. Every assurance in Figure 10.1 reduces to key custody and trust-list governance. An organisation with excellent credentials and casual key management has a presentational control.
• Retain originals. Chapter 5 noted that most organisations cannot produce the original of a disputed recording. Provenance is unavailable retrospectively, so retention policy is provenance policy.
• Do not label credentials “verified”. Say what is actually established: who signed it and what editing history is recorded. The authenticated lie is a certainty, and your interface copy decides whether it is also an endorsement.
• Prefer independently verifiable schemes. Where a provenance signal can be checked only by its vendor, you have outsourced the judgement rather than gained the capability.
• Push adoption collectively. The returns are a network effect, so sector bodies, standards work and procurement requirements move this and individual business cases do not.

Common misconceptions

Limitations and open questions

• The threshold argument is conceptual. Figure 10.2 asserts no adoption percentages, because credible cross-sector figures for credential prevalence in circulating media do not appear to exist. Measuring that prevalence would be the single most useful empirical contribution available here.
• Survival rates through real pipelines are unpublished. How often an embedded manifest reaches a viewer intact, by platform, is the number an implementer most needs and the number hardest to find.
• Trust-list governance is the real security boundary and is under-examined. Who may sign, who vouches for signers, and what happens on compromise or revocation are governance questions on which the cryptography depends entirely.
• Provenance has a privacy cost that this chapter does not resolve. A credential that records device, time and editing history can expose a source, a photographer in a hostile environment, or an activist. The specification provides for redaction, and the tension between proving origin and protecting origin is real, not a detail. An institution adopting credentials for its own material must decide what it does with material submitted by people at risk.
• No claim is made about current deployment scale. Figures circulating for marked-item counts and cross-vendor adoption were traced to secondary aggregators without method, and are excluded.

Teaching materials

Discussion questions

1. Why does provenance scale with your own output while detection scales with the adversary’s? What follows for budgeting?
2. If the returns on provenance are collective, what is the right instrument: standards, regulation, procurement, or something else?
3. Should a provenance interface distinguish a cryptographically bound manifest from one recovered by watermark lookup? What would the copy say?
4. Is a provenance signal verifiable only by its vendor worth having? Under what conditions?
5. How should a newsroom handle submitted material from a source whose safety depends on provenance not being recorded?

Assessment questions

1. Describe the C2PA manifest structure and explain why only one hard binding is permitted but many soft bindings are. (15 marks)
2. Explain the authenticated-lie failure mode and design interface copy that does not produce it. (15 marks)
3. Summarise the removal and spoofing results on watermarking and state what each implies for durable credentials. (20 marks)

Practical exercise

Establish who in your organisation would hold a content signing key, under what controls, and who could authorise its use. In most organisations this role does not exist, which means the control cannot currently be implemented regardless of budget.

Research assignment

Take ten images published by credential-signing organisations, retrieve them as a reader would through at least two platforms, and record how many still carry a valid manifest. Report the method and the attrition.

Seminar topic

“Provenance infrastructure protects institutions and endangers sources.” Argue either side.

Further reading

1. Coalition for Content Provenance and Authenticity. C2PA technical specification, version 2.x, and the Content Credentials explainer. spec.c2pa.org
2. ISO/DIS 22144, Authenticity of information — Content credentials. The standardisation route for the C2PA v2 architecture.
3. Saberi, M., Sadasivan, V. S., Rezaei, K., Kumar, A., Chegini, A., Wang, W., & Feizi, S. (2024). Robustness of AI-image detectors: Fundamental limits and practical attacks. ICLR. arXiv:2310.00076
4. Google DeepMind. SynthID, and the Nature publication of SynthID Text, October 2024. deepmind.google
5. Sadasivan, V. S., et al. (2023). arXiv:2303.11156 — for the text-watermark paraphrase result, C8.03.

Chapter synthesis

Provenance is the first control in this book whose effectiveness does not decay as generation improves, and the reason is structural rather than technical. Detection competes with a generator population that an adversary controls and continuously changes. A signature over your own output competes with nothing: the cryptography either holds or it does not, and it does not get worse when the models get better.

Three honest qualifications keep that from being a solution. The assurance is not uniform, because a manifest recovered by cryptographic hard binding and one recovered by watermark lookup after stripping carry materially different weight, and the published attacks on watermarking cut in both directions, removal and forgery. The credential answers a narrower question than its interface suggests, and the authenticated lie, arriving with a valid chain and a badge, is a certainty rather than a risk. And the value is a network effect: the organisation that signs first gains very little, because what eventually does the work is not the presence of a credential but the significance of its absence.

That last point is where the argument of this book turns outward. Nine chapters established that an institution acting alone cannot solve this at Layers 1 and 2, and Chapters 3, 5 and 9 showed that it can protect itself at Layer 4 with controls that are cheap, old and channel-based. Provenance is different in kind: it is the first layer where acting alone is genuinely insufficient and collective action is genuinely effective. Which is a question about mandates, liability and enforceable duties rather than about engineering, and that is Chapter 11.

The Legal and Regulatory Landscape

Transparency duties, likeness rights and evidence. Law has moved fastest on the question with the least purchase and slowest on the one with the greatest institutional exposure.

Three dates

Take the three legal questions synthetic media raises and ask where each one stands.

Who must disclose. Article 50 of the EU AI Act applies from 2 August 2026. Providers must mark AI-generated output in machine-readable form; deployers creating deepfakes must disclose that content has been artificially generated or manipulated. It was not postponed by the Digital Omnibus on AI, Regulation (EU) 2026/1744, which was published in the Official Journal on 24 July 2026 and in force from 27 July. That instrument did defer high-risk obligations: Annex III systems to 2 December 2027, and systems under Annex I sectoral product safety legislation to 2 August 2028. C11.01 Established

Whose likeness, and what harm. In the United States the TAKE IT DOWN Act was signed on 19 May 2025, making it unlawful to use an interactive computer service knowingly to publish an intimate visual depiction or a digital forgery of an identifiable person. Platform obligations took effect 19 May 2026: a notice-and-takedown mechanism, with properly reported imagery and known identical copies removed within 48 hours. The Federal Trade Commission has sent warning letters to 15 major platforms. C11.02 Established

Whether a court can rely on a recording. Proposed Federal Rule of Evidence 707, on machine-generated evidence, was published for comment in August 2025, with hearings in January 2026 and comments closing on 16 February. On 7 May 2026 the Advisory Committee voted to withdraw the published version and study a revised draft. In June 2026 the Standing Committee agreed and sent it back. Proposed Rule 901(c), which would let a court demand further authentication where a party credibly claims evidence was AI-fabricated, remains a study item. There is no rule and no effective date. C11.03 Established

That ordering is the chapter. The question that binds actors who were never the problem is in force. The question that binds intermediaries is in force and being enforced. The question that decides whether institutions can still rely on recorded evidence has no instrument at all.

Learning objectives

1. Distinguish the three legal questions and their very different compliance and effect profiles.
2. Explain why enforcement succeeds against intermediaries and fails against originators, using the documented cases.
3. Restate the function of transparency mandates in terms of the Chapter 10 threshold rather than as viewer warnings.
4. Describe the editorial-responsibility exemption in Article 50 and what it transfers.
5. Explain how proposed Rule 901(c) formalises the liar’s dividend as procedure.
6. Identify an organisation’s live legal exposure arising from Part II and Part III findings.

Reachability beats severity

Chapter 2 logged the enforcement outcome of the best-documented election case. Steve Kramer was identified, admitted commissioning the New Hampshire robocall, received a $6m FCC forfeiture and faced 13 felony and 13 misdemeanour counts. Reporting indicates neither the fine nor a subsequent $22,500 award was paid. C2.03

Set that beside the TAKE IT DOWN position. A statute aimed at platforms produced, within days of its compliance date, warning letters to fifteen named companies that have legal departments, revenue, jurisdictional presence and reputational exposure.

The difference between those two outcomes is not the severity of the law. The federal penalty in the first case was far larger than anything a warning letter represents. The difference is that one target was reachable and solvent and the other, though fully identified and domestic, effectively was not. Severity is what legislatures control; reachability is what determines collection.

What transparency duties are actually for

The standard criticism of disclosure mandates is that they bind only those who would have behaved well anyway. A compliant laboratory marks its output; a hostile actor does not, and faces no practical consequence for omitting a label on content that is already criminal. On its own terms the criticism is correct.

But it measures the wrong thing, and Chapter 10 supplies the correction. The value of provenance was never in the presence of a credential. It was in the eventual significance of its absence, which depends entirely on adoption, and which no individual organisation has an incentive to produce because the returns are collective.

A machine-readable marking duty on providers is the instrument that manufactures that adoption. Read that way, Article 50 is not a warning-label regime at all. It is adoption-forcing legislation whose function is to move the ecosystem toward the third panel of Figure 10.2, where unmarked output from an accountable source becomes a question worth asking. That is a more defensible rationale than the one usually offered for it, and a more modest one: the duty does not inform viewers so much as build the baseline against which absence will later be read. Analysis C11.04

Evidence: the liar’s dividend becomes procedure

Proposed Rule 901(c) would allow a court, where a party makes a credible claim that evidence was fabricated by artificial intelligence, to require further authentication. Consider what that does to the ordinary position of a party holding an authentic recording.

Today, a recording is authenticated by relatively light means. Under such a rule, an opponent’s credible assertion of fabrication can shift additional burden onto the party relying on genuine evidence. This is Chapter 2’s liar’s dividend arriving as civil procedure: the cost of the possibility of forgery is borne by whoever is telling the truth, because they are the one who must now prove more.

The rule is nonetheless probably necessary, and the dilemma is real rather than a drafting failure. Without such a provision, courts have no structured route to respond when fabrication is genuinely alleged. With it, every holder of authentic evidence inherits a new exposure. The practical escape is the one Chapter 10 described: an organisation that can produce an original with a hard binding and a documented editing history meets a raised authentication burden cheaply, and an organisation that cannot will meet it expensively or not at all.

Which means the single most consequential legal development in this field is a rule that does not yet exist, and the preparation for it is a records-retention and provenance decision that must be taken years before the rule lands.

Table 11.1 — The bottom-right cell is the one with a deadline nobody has set. Sources: C11.01 to C11.04.

Practical scenario

Executive implications

• Retain originals and provenance now, because the evidence rule has no date. This is the only item in this chapter with a lead time measured in years, and it is the one with no regulatory prompt to make you do it.
• Withdraw AI-text detection from decisions about people. Chapter 8 established a 61.22 per cent false-positive rate for non-native English writers against roughly five per cent for native writers. The evidence is published and the mechanism understood, which makes the disparate impact foreseeable, and foreseeability is where exposure begins.
• Mark your own output, and treat the duty as adoption rather than labelling. If Article 50 applies to you, compliance also buys the Chapter 10 benefit. If it does not, the benefit is still available.
• Use the editorial-responsibility route where you can. A named human accountable for publication is both a legal position and a better control than a label.
• Know the takedown mechanism before an incident. A 48-hour statutory clock is useful only to an organisation that already knows how to start it, for itself and for its people.
• Do not budget on deterrence. The best-documented penalty in this field, $6m against an identified and admitting defendant, appears uncollected.

Common misconceptions

Limitations and open questions

• This is not legal advice, and the selection is biased toward documentation. Two jurisdictions appear here because their instruments are well published. Likeness law in particular varies enormously, and several significant proposals, including US federal likeness bills, remain bills and are not treated as law.
• Statuses change, and this chapter is dated by construction. Article 50’s application date survived the Digital Omnibus; high-risk deferral shows such dates do move. Anyone relying on the dates here should re-verify them.
• Collection data is thin. C2.03 rests on reporting at a point in time, and no systematic data appears to exist on collection rates for synthetic-media penalties. That absence is itself a finding and would be worth a proper study.
• The transparency-as-adoption argument is analysis, not a stated legislative intent. Nothing cited here says Article 50 was designed to produce the Chapter 10 threshold effect. The claim is that this is its most defensible function, not that it was the drafters’ purpose.
• Enforcement against platforms has its own costs. A 48-hour removal duty with statutory penalties creates incentives toward over-removal, and the resulting harm falls on lawful speech. This chapter does not attempt to weigh that trade-off.

Teaching materials

Discussion questions

1. Why has law advanced fastest on transparency and not at all on evidence? Consider who lobbies for each.
2. Is the transparency-as-adoption argument a sufficient justification for a compliance cost on legitimate industry?
3. Article 50 exempts text with human editorial responsibility. Is that a loophole or the best part of the provision?
4. If Rule 901(c) shifts burden onto honest parties, should it be adopted anyway? What is the alternative?
5. Given the collection record, what should an organisation reasonably expect from legal remedy after an incident?

Assessment questions

1. Compare the three legal questions across the rows of Table 11.1 and justify which carries the greatest institutional exposure. (20 marks)
2. Explain why enforcement succeeded against intermediaries and failed against an identified originator. (10 marks)
3. Set out the evidentiary preparation an organisation should undertake before any new authentication rule takes effect. (15 marks)

Practical exercise

Pick one recording your organisation would rely on in a dispute. Establish whether an original exists, who holds it, whether it has been transcoded, and what provenance record accompanies it. In most organisations this exercise ends in under ten minutes with an unsatisfactory answer.

Research assignment

Trace one synthetic-media penalty from order to collection. Report what public records show about payment, and note how difficult the question was to answer.

Seminar topic

“Regulating disclosure was the easiest thing to do and the least useful; regulating evidence is the hardest and the most necessary.” Argue either side.

Further reading

1. Regulation (EU) 2024/1689 (AI Act), Article 50, and Regulation (EU) 2026/1744 (Digital Omnibus on AI). See the Commission’s FAQ on Article 50 transparency obligations. digital-strategy.ec.europa.eu
2. TAKE IT DOWN Act, Public Law, 19 May 2025. Congressional Research Service product LSB11314. congress.gov
3. Federal Trade Commission. (2026). Take It Down Act enforcement starts now. ftc.gov
4. Advisory Committee on Evidence Rules. Proposed FRE 707 and Rule 901(c), including the May 2026 withdrawal and continuing study.
5. Chesney, R., & Citron, D. (2019). California Law Review 107 — for the liar’s dividend that Rule 901(c) would formalise, see C2.03 and Chapter 2.

Chapter synthesis

Law is doing something in this field, and it is not what either its advocates or its critics usually describe. Transparency duties are in force and bind the compliant, which critics read as futility and advocates oversell as viewer protection. The better account is that a machine-readable marking duty is the only available instrument for manufacturing the adoption that makes a missing credential informative, because that benefit is collective and no firm will fund it alone. On that reading Article 50 is provenance policy, and its most interesting clause is the one that lets an organisation substitute a named accountable human for a label.

Likeness and harm law is in force too, and the comparison between its enforcement and the New Hampshire outcome is the practical lesson of the chapter. A $6m forfeiture against an identified, admitting defendant appears uncollected; warning letters to fifteen platforms produced immediate activity. Severity is what legislatures choose and reachability is what determines collection, so an institution should plan for intermediary remedies and treat deterrence of originators as unavailable.

The third question is the one with no instrument, no date and no constituency, and it is the one that will decide whether institutions can continue to rely on their own records. Proposed Rule 901(c) would let a credible allegation of AI fabrication raise the authentication burden, which transfers the cost of synthetic media onto whoever is telling the truth. It is probably necessary and it is certainly expensive, and the only cheap answer is a provenance and retention posture established before any dispute arises. That is the clearest instance in this book of a control whose deadline is set by litigation rather than regulation, which is to say by someone else entirely.

Against Figure 0.1, Layer 5 is now specified. It raises the floor, it reaches intermediaries, it will eventually reallocate evidentiary burden, and it will not protect any individual organisation from a competent attacker. What remains is the layer that has held in every documented case in this book, and the one that no statute will build for you. Chapter 12 assembles it.

Building an Organisational Defence Framework

Every control here is traced to the claim that justifies it, and every metric is one you can compute without knowing whether you have been attacked.

What eleven chapters did to the framework

This book opened with a five-layer framework and called it a thesis rather than a finding. It is worth being explicit about what the evidence did to it, because two of the five layers turned out to carry almost none of the weight originally assigned to them, and the sequence implied by drawing them as a stack turned out to be wrong.

The layers are not filters arranged in series, where material passes one to reach the next. They are independent failure points, and the attacker chooses which to engage. Arup was decided entirely at Layer 4 with Layer 2 never consulted. Slovakia was decided at Layer 4 and Layer 5, by a moratorium rule and a platform policy gap. The New Hampshire payload travelled by telephone, bypassing Layer 3 altogether.

Learning objectives

1. Rank candidate controls by the evidence supporting them rather than by convention or cost.
2. Identify commonly recommended measures that this book found no evidence for.
3. Construct indicators that can be computed without knowing whether an attack occurred.
4. Sequence implementation by lead time, recognising which control must start first despite paying last.
5. Assign each control to an accountable function, including the two that are usually unowned.
6. State what would falsify this framework.

The control set, ranked by evidence

Security frameworks are normally consensus documents. This one is not: every control below carries the claim that justifies it, and controls with no supporting evidence in this book are listed as such rather than quietly included.

Table 12.1 — Nine controls, each traceable to a ledger claim. Note that the top two are old, cheap and procedural, and that one of the nine is a withdrawal. Source: author’s synthesis.

Measure the controls, not the threat

Chapter 7 established that content authenticity has no feedback channel: a successful evasion is indistinguishable from authentic material, so a deployed detector’s real-world failure rate is unmeasurable in principle from the defender’s side. That finding has a constructive consequence for framework design that is easy to miss.

If you cannot measure the threat, build the programme on indicators of your own controls, all of which are computable without knowing whether anyone has attacked you. This is the opposite of the usual dashboard, which counts incidents and flags and therefore reports the attacker’s activity rather than your readiness.

Sequencing, by lead time rather than by priority

Ranking controls by evidence tells you what matters. It does not tell you what to start first, because the control with the longest lead time is not the one with the highest rank.

Start immediately, pays in years. Originals and provenance. Retention and capture decisions determine what you can produce in a dispute three years hence, the evidentiary rule that will raise the burden does not yet exist and so creates no prompt, and nothing can be reconstructed retrospectively. This is the only item whose deadline is set by litigation rather than by regulation, which is to say by someone else.

First ninety days. Rewrite verification guidance into the three explicit properties. Count your single-channel authorisation paths. Name the people who may issue a denial without further sign-off, and measure the rebuttal window as it actually stands at two in the morning. Withdraw AI-text detection from decisions about people. Four of these are policy, cost almost nothing, and address the two highest-ranked controls plus the one withdrawal.

First year. Signing for your own output, with key custody assigned. Red-teaming for any detector in service. Population-level monitoring where synthetic identity is a live risk. Sector and procurement engagement on provenance adoption, because those returns are collective and nothing you do alone produces them.

Practical scenario

Executive implications

• The first ninety days should contain no procurement. The two highest-ranked controls are policy wording and process design, and the sixth is a withdrawal. Anything that must be bought ranks fifth or lower.
• Report readiness, not incidents. Replace flag counts and “no known evasions” with the six indicators in Figure 12.2. The first is a guaranteed observation and misleads boards in a specific, correctable way.
• Start retention and provenance now, out of sequence. It ranks third and must begin first, because its lead time is years and no regulator will prompt you.
• Assign the two unowned controls explicitly. A rebuttal window with no owner has never been measured, and a signing key with no custodian makes provenance unimplementable.
• Treat provenance adoption as a collective project. Your own signature buys little until absence is informative, so sector bodies and procurement requirements are the instrument, not internal business cases.
• Expect no help from deterrence, and plan for intermediary remedies. Know the statutory takedown route before you need it, for the organisation and for your people.

Common misconceptions

Limitations and what would change the conclusions

• The documented case base is six corporate incidents and a handful of election cases. That is what the public record supports, and organisations that lost money quietly are absent. The uniform result on detection should be read as a strong absence of evidence, not as proof.
• The ranking is argued, not validated. No controlled comparison of these controls exists, and none is likely. A reader who weighs the cases differently can reasonably reorder ranks three to nine; ranks one and two rest on the clearest evidence in the book.
• What would falsify the framework. A documented case in which detection, rather than process, determined the outcome would move Layer 2 up. A measured deployment showing stable precision at low prevalence would reopen Chapter 4. Evidence that attack volume against mid-value targets has not risen would withdraw the agentic argument of Chapter 9. Each of these is a specific observation, and the framework should change if any of them appears.
• Two ledger claims remain Emerging and cannot be improved by further reading. C2.03 rests on reporting at a point in time; C3.02 rests on an insurer’s account of a claim against an unnamed company. They are marked rather than scheduled.
• The book takes no position on several real questions. Over-removal harms from statutory takedown, the privacy cost of provenance to sources at risk, and the disparate impact of coordination analysis are each raised and left unresolved, because resolving them requires value judgements rather than evidence.

Teaching materials

Discussion questions

1. The two best-evidenced controls require no new technology. Why does the market offer mostly technology?
2. Should a framework ever include a measure with no supporting evidence? Under what conditions?
3. Which of the six indicators would your organisation find hardest to compute, and what does that difficulty reveal?
4. Who should own the rebuttal window, given that it spans communications, legal and the executive office?
5. If detection failure is unobservable, is deploying detection ever defensible outside a closed funnel?

Assessment questions

1. Reconstruct Table 12.1 for your own organisation, with owners and current status for each control. (25 marks)
2. Explain why the framework is sequenced by lead time rather than by rank, with the worked example. (10 marks)
3. State three observations that would falsify parts of this framework and explain what each would change. (15 marks)

Practical exercise

Compute all six indicators in Figure 12.2 for your organisation this week. Record which ones you could not compute at all. That list is your actual starting position, and it is more useful than any maturity score.

Research assignment

Take any published synthetic-media guidance and score each of its recommendations against the evidence standard used here: documented case, primary source, or neither. Report the proportion in each category.

Seminar topic

“Evidence-ranked frameworks are better than consensus frameworks, and that is why nobody publishes them.” Argue either side.

Further reading

1. Federal Bureau of Investigation, IC3. (2025). Public service announcement I-051525-PSA. The clearest official statement of the top-ranked control, C9.01.
2. Financial Crimes Enforcement Network. (2024). Alert FIN-2024-Alert004. fincen.gov — the population-level indicator model behind control 8.
3. Coalition for Content Provenance and Authenticity. C2PA technical specification. spec.c2pa.org — for controls 3 and 5.
4. Liang, W., et al. (2023). Patterns 4(7). cell.com/patterns — the basis for the withdrawal at rank 6.
5. Carlini, N., & Farid, H. (2020). CVPR Workshops. openaccess.thecvf.com — why control 7 is necessary wherever control 9 is used.

Chapter synthesis, and the book’s

The argument of this book can be stated in four sentences. Generation is not controllable, and the cost of imitating a person has fallen to three seconds of public audio. Detection is an instrument with a narrow envelope, five independent reasons its published performance does not transfer, an unpublished expiry date, and no means of telling you when it has failed. Provenance is the only control whose effectiveness does not decay as generators improve, and its value is collective rather than individual, which is why mandates matter more than business cases. And in every documented case where money moved or did not move, the outcome was determined by institutional process, by whether one channel could both request and approve, and by whether anybody had to be reached on a path they did not choose.

That last finding is the one worth carrying. It is also the least satisfying, because it says the answer to a novel technological threat is mostly old administrative hygiene: a question whose answer is not published, a number from your own directory, a second approver contacted separately, a named person permitted to speak at two in the morning without waiting for sign-off, and a retained original of anything you might one day have to prove. None of it is interesting. All of it held.

The uncomfortable corollary is that the industry grew up around the layer that does not hold. Detection attracts the research, the procurement budgets and the vendor claims, and it is the layer where the evidence is weakest, the feedback is absent and the harms of error fall on identifiable people, overwhelmingly on the ones least able to contest an accusation. A twelve-fold disparity in false positives against non-native speakers is not a rough edge on a working tool. It is what the tool does.

This book has tried to be wrong in visible ways rather than right in vague ones. Forty-seven claims carry identifiers, sources and stated limits; two remain marked as weak because their sources cannot be improved; the three analytical claims that do most of the work are labelled as analysis rather than dressed as findings; and Chapter 9’s prediction about agentic attack volume is written so that it can be shown false. If the framework in Table 12.1 is wrong, it should be wrong in a way a reader can demonstrate. That is the most useful property a book on this subject can have, in a field whose central difficulty is that nothing tells you when you have been fooled.