Chapter 5 of 1212 min read

Voice Cloning and Audio Deepfake Detection

Three seconds is enough to clone a voice. The channel that carries nearly every real attack is also the channel that destroys the evidence a detector needs.

Three seconds

In January 2023 Microsoft researchers published VALL-E, which treats speech synthesis as language modelling over discrete audio codec tokens. Pretrained on 60,000 hours of English speech, it synthesises a target speaker from a three-second enrolled recording of a speaker it has never heard, and preserves the emotion and acoustic environment of that prompt. C5.01 Established

This resolves the claim Chapter 1 left open. The cost of cloning a voice is no longer a training run against a corpus of your target; it is three seconds of audio. Every executive who has given a conference talk, every official who has answered a reporter, every manager who has recorded a voicemail greeting has already published the enrolment prompt.

That fact alone would make this a short chapter: voice can no longer function as an identity claim, and Chapter 3 showed that identity claims were never where losses were decided anyway. But audio detection deserves a fuller account than that, because it is where the detection field has its longest and most honest evidence base, and where that evidence points to something more specific and more awkward than in the video case.

The awkward finding, stated up front

Audio spoofing countermeasures depend heavily on fine spectral detail, much of it at higher frequencies. Narrowband telephony discards almost all of it. So the channel through which nearly every documented voice attack has actually arrived, the telephone call and the compressed voice note, is the channel in which countermeasures perform worst. Detection is weakest exactly where it is needed most, and this is a property of the channel rather than a deficiency anyone can engineer away. Analysis

Learning objectives

State the current cost of cloning a voice and the implication for voice as an authentication factor.
Explain how narrowband and lossy coding remove the signal that audio countermeasures rely on.
Interpret ASVspoof results correctly, distinguishing closed-condition challenge performance from field performance.
Describe what the in-the-wild evaluation established and the limits of its headline figure.
Decide where in an audio pipeline detection should be applied, if at all.
Specify the controls that hold for voice attacks without requiring any judgement about the audio.

Where the evidence lives, and where it goes

An authentic recording and a synthetic one differ in ways that are mostly small and mostly high. Vocoder and codec-decoder output tends to leave traces in the upper spectrum, in the fine structure of harmonics and noise, and in the statistical regularity of phase. A countermeasure trained on full-band studio audio has a great deal of that material to work with.

Now send the same audio through a telephone network. Classical narrowband telephony encodes roughly 300 to 3,400 Hz at an 8 kHz sampling rate under ITU-T G.711. C5.04 Established Everything above 4 kHz is simply absent. Wideband voice over IP does better, up to around 7 kHz, but applies perceptual coding designed to discard anything a listener will not miss, which is a reasonable description of the artefacts a detector is hunting. Add packet loss concealment, automatic gain control and re-encoding by a messaging platform, and the residue that distinguished the two recordings has been attenuated by processes that are entirely legitimate and completely routine.

Figure 5.1 — The attack channel and the evidence-destroying channel are the same channel. Bandwidths are drawn to scale on a linear frequency axis. The shaded region indicates where spoofing residue is commonly reported, not a measured distribution. Sources: ITU-T G.711 (C5.04); author’s analysis.

What the challenge record establishes

Audio has a better evidence base than video, because the ASVspoof series has run repeated community challenges on shared data since 2015, with published overviews. ASVspoof 5, reported in 2024, is the most informative edition: crowdsourced speech from almost two thousand speakers, an order of magnitude more than earlier editions, adversarial attacks included for the first time, and 53 participating teams. Many submissions performed well. The overview also records that performance degrades under adversarial attacks and under the application of neural encoding and compression schemes. C5.03 Established

That second sentence is the one to carry away. It is not a complaint from a critic; it is the challenge organisers reporting that the conditions which make a deployment realistic are the conditions in which the field’s best systems lose ground.

The complementary result comes from outside the challenge setting. Müller and colleagues, at Interspeech 2022, assembled 37.9 hours of found audio of public figures, of which 17.2 hours were deepfakes, and evaluated existing methods on it. Their finding, in the paper’s own words, is that related work performs poorly on such real-world data, with performance degradation of up to one thousand per cent. C5.02 Established

Evidence source	What it establishes	What it does not
ASVspoof 5 2024 challenge overview	Strong closed-condition performance is attainable at scale, across ~2,000 speakers and 53 teams; and that performance degrades under adversarial attack and neural codecs	Field performance. Challenge data is shared in advance and the attack set is bounded.
In-the-wild evaluation Müller et al., Interspeech 2022	That methods trained and tested on curated data perform poorly on found audio of public figures, with degradation reported up to one thousand per cent	A universal multiplier. The figure is relative to the specific baselines evaluated, at one point in time.
VALL-E arXiv:2301.02111, 2023	That a three-second prompt suffices for zero-shot cloning, with emotion and acoustic environment carried over	Anything about commercial tooling quality, which is not the same question and is not measured here.

Table 5.1 — The three sources this chapter rests on, with their limits stated beside them. As in Chapter 4, no current equal error rate is asserted. Sources: claims C5.01 to C5.03.

3 sCloning promptEnrolled recording sufficient for zero-shot synthesis of an unseen speaker.

60,000 hPretraining corpusThe cost sits with the model builder, not with the attacker.

17.2 hIn-the-wild deepfakesWithin a 37.9-hour found-audio dataset of public figures.

3.4 kHzTelephone ceilingAbove which the spectrum the detector wants does not arrive at all.

Case study: the attack channel paradox

Case study 5.1 · analysis over established cases

Take the voice incidents already established in this book and record the channel each one used.

UK energy firm, 2019: Telephone call. Narrowband. C3.02
Retool, 2023: Telephone call following an SMS pretext. Narrowband. C3.03
LastPass, 2024: WhatsApp voice messages. Compressed, perceptually coded. C3.04
Ferrari, 2024: WhatsApp messages then a call. Compressed, then narrowband. C3.05
WPP, 2024: Voice clone inside a Teams meeting. Perceptually coded, with packet-loss concealment. C3.05

Not one arrived as a file. Every single one arrived through a real-time or messaging voice channel that had already removed much of the spectrum a countermeasure would want, and in several cases the audio was never retained at all. Analysis

Why this is more than an inconvenience. It means the deployment most people imagine, an audio detector sitting on the corporate telephone system or screening voice notes, is operating in close to the worst available conditions on close to the least informative copy of the signal. It also means the research setting and the operational setting are separated by a transformation that nobody is doing anything wrong by applying.

What follows for control design. Exactly what Chapter 3 established, and from a second independent direction. The control that holds for a voice attack does not involve the audio: a question whose answer the caller cannot obtain, a callback to a number from your own directory, an approval that cannot be satisfied through the channel that made the request. None of these degrade with bandwidth.

Voice biometrics as an authentication factor

Voice enrolment was adopted by banks and helpdesks on the reasoning that a voice is hard to imitate. On the current evidence it should be treated as a public identifier rather than a secret: it is broadcast continuously, cannot be rotated after compromise, and the acquisition cost for an attacker is three seconds.

That does not make it worthless. It makes it a convenience feature and a weak signal to combine with others, which is a very different thing from the factor that releases funds or resets a credential. Any process in which a matched voiceprint alone authorises an action should be rewritten, and the test from Chapter 3 applies unchanged: can the channel that carried the request also satisfy the approval?

Practical scenario

Illustrative scenario · not a documented incident

Your contact centre receives 40,000 calls a month on a narrowband trunk. A vendor proposes real-time synthetic-speech screening, quoting an equal error rate near one per cent on a public benchmark, and recommends that flagged calls be routed to a supervisor who informs the caller that the call has been identified as potentially fraudulent.

Work through: what the benchmark condition was and how it differs from a narrowband trunk; what Chapter 4’s precision arithmetic gives at your actual prevalence; and what the supervisor’s script does to the large majority of flagged callers who are genuine customers. Then specify the one control you would buy instead, with its cost.

Executive implications

Remove voice as a sole or primary authentication factor. Three seconds of your executives’ recorded speech is already public. This is a policy change, not a procurement.
Do not expect benchmark behaviour on telephone audio. The published conditions and your trunk are separated by a bandwidth reduction that removes most of the detector’s evidence.
Detect on the best copy, as early as possible. If audio detection is in scope, apply it to the highest-fidelity recording available before transcoding, never to a downstream re-encode.
Retain original audio where law allows. Most organisations cannot produce the original of a disputed recording, which forecloses both detection and provenance. Chapter 10 takes this further.
Buy the channel-independent control first. Out-of-band verification works at 3.4 kHz, at 48 kHz, and on a transcript. It is also cheaper.

Common misconceptions

Misconception 1 · “Audio is a simpler signal, so detection should be easier”

Lower dimensionality cuts both ways. A face has substantial spatial redundancy, so some artefacts survive heavy compression. A few seconds of narrowband speech has very little margin, and the band where spoofing residue is commonly reported is the band the channel removes first. Simplicity here means less evidence, not an easier problem.

Misconception 2 · “Equal error rates near one per cent mean the problem is solved”

Those figures come from closed conditions on shared data with a bounded attack set. The same literature reports degradation under adversarial attack and neural codecs, and the in-the-wild evaluation reports degradation of up to one thousand per cent against found audio. A benchmark equal error rate is a statement about a dataset, and Chapter 4’s precision arithmetic still has to be applied on top of it.

Misconception 3 · “Asking the caller to say a passphrase defeats cloning”

A passphrase is content, and content is exactly what a text-to-speech system produces on demand. Challenge-response only helps when the challenge requires knowledge the attacker lacks, not when it requires speech the attacker can synthesise. The Ferrari question worked because of the answer, not because of the voice that delivered it.

Limitations and open questions

The thousand per cent figure is relative and bounded. It is the paper’s own characterisation of degradation against the specific baselines it evaluated in 2022. It should not be quoted as a general multiplier, and it is not evidence about any current system.
No current equal error rate is asserted. As in Chapter 4, the durable finding is structural. Challenge results improve annually and the gap between closed and field conditions has proved more persistent than any single score.
Where spoofing residue sits is reported, not quantified here. Figure 5.1 marks a region qualitatively. A rigorous version would state, per vocoder family, how much discriminative information survives a given codec, and that measurement is not consistently published.
Real-time cloning is Chapter 9. This chapter treats synthesis as something produced before the call. Interactive cloning changes the operational picture and is handled with the other emerging threats.
Adversarial evasion is Chapter 7. ASVspoof 5 includes adversarial attacks for the first time, and the degradation it reports under them belongs to that discussion.

Teaching materials

Discussion questions

If a voice is a public identifier rather than a secret, what processes in your organisation currently treat it as a secret?
Why does the same compression that makes a call intelligible make it harder to assess? Is there any way to have both?
The challenge organisers themselves report degradation under codecs. Why has that not changed procurement behaviour?
What would it take to retain original-fidelity audio for disputed calls, and what are the privacy consequences of doing so?
Ferrari’s executive asked a question rather than listening harder. Why is that the general solution rather than a lucky one?

Assessment questions

Explain, in terms of bandwidth and coding, why audio countermeasures degrade on telephone channels. (10 marks)
Given a benchmark equal error rate and a prevalence, compute the precision of a contact-centre deployment and state the consequence for genuine callers. (15 marks)
Argue for or against retaining voice biometrics in a customer-authentication flow, given a three-second cloning cost. (20 marks)

Practical exercise

Find three seconds of clean public speech for a named executive in your own organisation, using only public sources, and time how long the search takes. Do not synthesise anything. The finding is the search time, and in most organisations it is under a minute.

Research assignment

Take one published audio countermeasure with available code and evaluate it on the same audio before and after narrowband coding. Report the change. State clearly what your experiment does and does not establish.

Seminar topic

“Voice biometrics should be withdrawn from financial authentication entirely.” Argue either side, costing the customer-experience consequences.

Chapter synthesis

The audio case sharpens the argument of Part II rather than repeating it. Chapter 4 found two reasons a published detector score does not transfer: the test distribution is drawn from the training generators, and precision collapses at institutional prevalence. Audio adds a third, and it is the most structural of the three. The signal a countermeasure depends on is concentrated where the communication channel is designed to discard information, and the channels that do most of that discarding are precisely the channels through which every documented voice attack has arrived.

This is not a gap that better models close, because nothing is malfunctioning. Narrowband coding is working as specified. Perceptual coding is doing the job it was designed for. An organisation screening telephone audio for synthetic speech is asking a detector to read evidence that the telephone system has already, correctly, thrown away.

Against Figure 0.1, Layer 2’s envelope narrows again, and Layer 1, generation, has moved decisively: three seconds and a public recording. What has not moved is Layer 4. The question Chapter 3 ended on, whether the channel that carried the request can also satisfy the approval, is bandwidth-independent, works on a transcript, and would have stopped every case in Table 3.1. Chapter 6 turns to why the benchmarks that produce these scores are constructed the way they are, and what it would take to build one that predicted field behaviour.