Chapter 6 of 1212 min read

Benchmarking, Datasets, and Cross-Domain Generalization

Detectors expire for reasons that have nothing to do with attackers, and the benchmarks that certify them cannot see it happening.

The paper that found the signal and published its cure

In 2020 Durall, Keuper and Keuper showed that the up-sampling operations inside generative convolutional networks fail to reproduce the spectral distribution of natural images. Transposed convolutions add high-frequency noise; interpolation-based up-sampling leaves high frequencies missing. They noted that these spectral distortions can be used to detect generated data, reporting up to 100 per cent accuracy on public benchmarks. C6.01 Established

In the same paper they proposed a spectral regularisation term for the generator’s training objective, which removes the high-frequency error, stabilises training and improves output quality.

Read those two paragraphs together. One publication identified a detection signal good enough for near-perfect benchmark accuracy, and supplied the modification that eliminates it, offered not as an evasion technique but as an improvement to image quality. Any generator author who adopted it for the stated reason, better pictures, destroyed the detector as a side effect and would have had no idea they had done so.

This is the mechanism this chapter is about. Detection capability decays for reasons entirely unconnected to anyone attacking a detector. Deliberate evasion exists and is the subject of Chapter 7. Obsolescence is different: it is automatic, it is driven by ordinary progress in generation, and it is invisible to the measurement apparatus the field uses to certify progress.

Learning objectives

Distinguish detector obsolescence from adversarial evasion, and explain why the first is the larger operational problem.
Identify the axis along which detection benchmarks have scaled and the axis that actually governs transfer.
Explain why a cross-dataset performance drop does not identify its own cause.
State what a transfer-honest evaluation must report, and audit a published paper against that list.
Describe the comparability problem that unified benchmarking frameworks were built to address.
Decide what evidence would justify trusting a detector on content it has not seen.

The axis the field did not scale

Look at how the reference datasets grew. FaceForensics++, in 2019, contained 1,000 authentic videos and four manipulation methods. Celeb-DF raised synthesis quality. DFDC raised scale, added organic video and introduced a genuine black-box condition. Each was a real contribution and each made the data harder.

Now ask what each one varied. Mostly identities, resolution, compression, lighting, pose, and the sheer number of clips, which is to say the content and the conditions. The number of distinct generator families stayed small throughout. That matters because the evidence says generator family is the variable that governs whether a detector transfers, and content diversity is not a substitute for it.

Corvi and colleagues demonstrated the point directly at ICASSP 2023: detectors built for GAN-generated imagery struggle on images from diffusion models, and, in the stronger result, even when retrained on diffusion output they often fail to generalise across different diffusion models. C6.02 Established A detector can be trained on a vast, varied, high-quality corpus and still be defeated by a change of architecture it has never met.

Figure 6.1 — Fifteen years of harder datasets, almost all of it on the vertical axis. Content and condition diversity rose substantially. Generator family diversity, which the evidence identifies as the variable governing transfer, rose very little. ASVspoof 5 is the partial exception, and reports degradation under exactly the conditions it added. Source: author’s arrangement; see limitations. Analysis C6.04

Why a cross-dataset drop does not explain itself

The standard robustness experiment is to train on dataset A and test on dataset B, then report the fall. It is better than not doing it, and it is much weaker evidence than it looks, because A and B differ simultaneously in generator family, synthesis quality, compression history, resolution, capture conditions, demographics and labelling policy. A drop from 0.95 to 0.62 AUC is a fact about two datasets. It attributes nothing.

This is the methodological gap that unified benchmarking was created to close. DeepfakeBench, presented at NeurIPS 2023 by Yan and colleagues, was motivated by exactly this: data-processing pipelines were not uniform, so detection models were not receiving comparable inputs; experimental settings differed noticeably; and evaluation strategies and metrics were not standardised. It supplies a unified data-management layer, a common implementation framework for 15 detection methods across 9 datasets, and standardised protocols. C6.03 Established

The existence of that contribution is itself a finding worth stating plainly. A field in which published numbers are not comparable to one another had been reporting progress against those numbers for several years.

What published evaluations usually report	What a deployment decision needs
AUC or accuracy on a held-out split of the same datasets	Performance on generator families held out of training entirely
A single aggregate figure	Per-generator breakdown, so the weakest case is visible rather than averaged away
The ranking on one test condition	Rank correlation between familiar and unfamiliar conditions, which DFDC showed can invert
Scores at an unstated or implicit threshold	Calibration, plus the operating threshold and the precision it yields at your prevalence
Clean, curated media	Results after the actual degradation chain: recompression, re-encoding, cropping, screen capture, narrowband coding
Nothing about the generator population’s drift	An expiry expectation, and what observation would reveal that expiry has occurred

Table 6.1 — The right-hand column is a procurement specification. Very few published evaluations supply more than two of its rows. Source: author’s synthesis of the issues raised by C4.02, C6.02 and C6.03.

100%Benchmark accuracyReported from a spectral signal the same paper showed how to remove.

15 / 9DeepfakeBench scopeMethods and datasets brought under one comparable protocol, 2023.

4Generator families in FF++The reference training set for much of the field.

2Axes, one sampledContent diversity grew; generator diversity, which governs transfer, did not.

Case study: a detection signal with a published expiry date

Case study 6.1 · documented, primary source

The finding: Common up-sampling methods leave generative models unable to reproduce natural spectral distributions: transposed convolution adds high-frequency noise, interpolation leaves high frequencies absent
Detection value: The resulting distortion supports detection of generated media, with up to 100 per cent accuracy reported on public benchmarks
The cure, in the same paper: A spectral regularisation term added to the training objective, which removes the high-frequency error while improving training stability and output quality
Source: Durall, Keuper & Keuper (2020), CVPR. Established

What makes this the cleanest available illustration. The incentive to adopt the fix is not evasion. It is picture quality and training stability, which is what generator authors are paid to improve. A detection signal whose removal is a by-product of the generator community’s own objectives is not a durable control, however well it scores.

The general pattern. A signal is discovered, published, and adopted by detectors. Publication tells generator authors precisely which property of their output is anomalous. They remove it, usually for quality reasons. The detector expires, silently, and the benchmark continues to report the old number because the benchmark’s generators have not changed.

Figure 6.2 — Obsolescence is a cycle, and the benchmark sits outside it. No attacker is required at any stage. Source: author’s analysis; the worked instance is C6.01.

Practical scenario

Illustrative scenario · not a documented incident

You procured a detector eighteen months ago on the strength of a strong published evaluation and an independent test against your own archive. It has been in production since, flagging a stable one to two items a week, and nobody has complained. Your supplier has shipped three model updates. The generator landscape has changed twice.

Answer honestly: what observation currently available to you would distinguish “still working” from “expired, and now flagging only the easy historical cases”? If the answer is none, the detector has been unmonitored for eighteen months, and a stable flag rate is consistent with both states.

Executive implications

Treat a detector as a perishable with an unknown expiry date. It does not fail loudly. It degrades while its flag rate looks normal, so plan a re-validation cadence rather than waiting for a signal that will not come.
Hold back your own test set, and never share it. A private set of material from generators your supplier has not trained on is the only instrument that tells you anything a benchmark cannot.
Require a per-generator breakdown. An aggregate figure hides the weakest case, which is the one an attacker will choose.
Ask what would make this detector obsolete. A supplier who cannot answer has not thought about the lifecycle in Figure 6.2.
Do not interpret a stable flag rate as evidence of health. It is equally consistent with a detector catching only the generators it already knew.

Common misconceptions

Misconception 1 · “More and harder data will fix generalisation”

The datasets did get larger and harder, and the transfer gap persisted. The growth was mostly on the content and condition axis, while the variable that governs transfer is generator family. Corvi and colleagues show the limit case: retraining on diffusion output still does not reliably generalise across different diffusion models. Scaling the wrong axis buys robustness to the wrong thing.

Misconception 2 · “Cross-dataset evaluation measures generalisation”

It measures a difference between two datasets that vary simultaneously in generator, quality, compression, resolution and demographics. The resulting number is real and uninterpretable as a cause. Only holding out generator families while controlling the rest attributes the drop to the thing you care about.

Misconception 3 · “Detectors decay because attackers adapt”

Some do. But the dominant mechanism needs no adversary. Spectral regularisation was proposed to improve output quality and training stability, and it removes the artefact that supported near-perfect benchmark accuracy. Most detector decay is collateral damage from other people improving their products, which is why it is continuous rather than event-driven.

Limitations and open questions

Figure 6.1 is a judgement, not a measurement. There is no accepted metric for generator family diversity, so the positions express an argument about direction of travel. Constructing such a metric would be a genuinely useful contribution and would allow the claim to be tested or refuted.
Leave-one-generator-out is necessary and not sufficient. Generator families are not cleanly separable: architectures share components, training data overlaps, and many deployed systems are proprietary and cannot be enumerated at all. A held-out protocol improves the evidence without closing the gap.
No current benchmark results are quoted. As in Chapters 4 and 5, the finding here is about measurement structure. A current leaderboard figure would date immediately and would also be the kind of number this chapter argues against relying on.
The lifecycle in Figure 6.2 is not quantified. How long a published signal survives in practice is an empirical question, and a study tracking signal lifetimes across generator releases does not appear to exist.
Deliberate evasion is excluded here by design. Chapter 7 adds an attacker who is specifically targeting the detector, which is a different and faster failure mode.

Teaching materials

Discussion questions

Durall and colleagues published both a detection signal and the modification that removes it. Should they have? What follows for publication norms in this field?
Why has the community scaled content diversity so much more readily than generator diversity? Consider cost, licensing and credit.
If a stable flag rate cannot distinguish a working detector from an expired one, what instrumentation would?
Is a benchmark whose numbers are not comparable across papers worse than no benchmark?
Which rows of Table 6.1 could your organisation actually require from a supplier, and which would be refused?

Assessment questions

Explain why a drop from 0.95 to 0.62 AUC across datasets does not attribute a cause, and design an experiment that would. (15 marks)
Distinguish obsolescence from evasion, with a worked example of each. (10 marks)
Write a leave-one-generator-out evaluation protocol, and state what it still cannot establish. (20 marks)

Practical exercise

Take any published detection paper and score it against the six rows of Table 6.1. Record the score. Then do the same for the marketing material of a commercial detector, and compare.

Research assignment

Pick one published detection signal and trace what happened to it: when it was reported, which detectors adopted it, and whether subsequent generator releases retained or removed the underlying property. Report how far you could get with public information.

Seminar topic

“Detection research should not publish exploitable signals, for the same reasons vulnerability disclosure is coordinated.” Argue either side.

Chapter synthesis

Part II has now produced four independent reasons a published detector score does not transfer to a deployment, and only one of them is about machine learning being hard. Chapter 4: the test distribution is drawn from the training generators, and the ranking can invert. Chapter 4 again: precision collapses at institutional prevalence, as arithmetic. Chapter 5: the operational channel removes the evidence, by design and correctly. Chapter 6: the signal expires because generator authors improve their products, and the benchmark cannot observe the expiry because its generators are fixed.

The fourth is the one that should change behaviour, because it is the only one that keeps acting after procurement. A detector bought on sound evidence, validated on your own archive and performing visibly well can be substantially obsolete eighteen months later with no outward sign, and the single most common health indicator, a stable flag rate, is consistent with both outcomes. Durall and colleagues showed the whole mechanism inside one paper: find the anomaly, publish it, and supply the fix that quality-conscious engineers will adopt for their own reasons.

This closes the last of the open literature-direction claims in the ledger. Chapter 1 asserted that early detectors exploited generator-specific up-sampling traces and generalised poorly, and marked it Emerging pending this chapter. It is now pinned: Durall for the trace, Corvi for the failure to transfer across generator families, including within diffusion. Against Figure 0.1, Layer 2 can be stated precisely at last. Detection is a measurement of a relationship between a detector and a generator population that someone else controls and continuously changes for reasons of their own. It is useful inside the envelope Chapter 4 specified, it expires on a schedule nobody publishes, and it was never the layer that was going to hold. Chapter 7 removes the last generous assumption by introducing an adversary who is aiming at the detector on purpose.