Chapter 8 of 1212 min read

LLM-Generated and Multimodal Disinformation

Text detectors do not measure provenance. They measure fluency, and that is why their errors land on the same people every time.

Sixty-one per cent

Liang and colleagues, publishing in Patterns in 2023, ran seven commercial GPT detectors over TOEFL essays written by non-native English speakers. The average false-positive rate was 61.22 per cent. All seven detectors unanimously flagged 19.8 per cent of those human-written essays as machine-authored, and at least one detector flagged 97.8 per cent of them. For essays by native English writers the false-positive rate was roughly five per cent. C8.02 Established

Then the detail that settles what these tools are actually measuring. The authors reduced the misclassification by using ChatGPT to enrich the vocabulary of the TOEFL essays, which brought the average false-positive rate down from 61.3 to 11.6 per cent.

Putting an LLM through your essay makes it look less like an LLM wrote it. No instrument measuring provenance could behave that way. These detectors estimate how statistically unsurprising a text is, usually through perplexity, and a writer with a narrower range of English expression produces exactly the signature they are built to flag. The false positives are not noise around a working measurement. They are the measurement.

Learning objectives

Explain why text offers no generator artefact of the kind images and audio provide, and what detectors substitute for one.
State the measured disparity in false-positive rates between native and non-native English writers.
Describe the effect of recursive paraphrasing on watermarked and non-watermarked detection.
Summarise the theoretical limit on detection as human and machine text distributions converge.
Distinguish the generation question from the coordination question, and say which is actionable.
Decide where AI-text detection may and may not be used in decisions affecting people.

Why text is different

Part II rested on a physical fact: a generator leaves traces in a sampled signal. Up-sampling disturbs a spectrum, a vocoder leaves residue in harmonics, a composite leaves a seam. Those are properties of a process acting on a continuous medium, and even when they are perishable they are real.

Text has nothing equivalent. It is discrete, short, and carries no sampling process to fingerprint. There is no pixel grid and no spectrum. What remains is the statistical shape of the word sequence, which is to say its style. And style is also the thing that distinguishes one human writer from another, so any detector operating on it necessarily conflates two different notions of unusual: produced by a machine, and written by an atypical human.

Who counts as atypical is not random. It is the second-language writer, the writer using a formulaic register because that is what their training taught, the writer with a smaller active vocabulary, the writer working in a technical template. The measured twelve-fold disparity is the predictable consequence of the design, not a bug awaiting a patch.

Figure 8.1 — A twelve-fold disparity, with the mechanism visible in the third bar. An instrument whose error rate falls when you add more machine involvement is not measuring machine involvement. Source: C8.02.

26%OpenAI classifier TP rateTrue positives on its own challenge set, with 9% false positives. Withdrawn July 2023.

12×Bias ratio61.22% false positives for non-native writers against roughly 5% for native writers.

99 → 15Watermark TPRAt 1% false-positive rate, after five rounds of recursive paraphrasing.

97.8%Flagged at least onceOf human-written TOEFL essays, by at least one of seven detectors.

Case study: the maker withdraws its own detector

Case study 8.1 · documented, first-party

Tool: OpenAI’s AI Text Classifier, released January 2023
Reported performance: On a challenge set of English texts, it correctly identified 26% of AI-written text as likely AI-written, and incorrectly labelled human-written text as AI-written 9% of the time
Outcome: Withdrawn on 20 July 2023, citing a low rate of accuracy, with low reliability on short and paraphrased text among the stated reasons
Source: OpenAI’s own announcement and withdrawal. Established C8.01

Why this is the strongest available evidence on text detection. It is first-party, it is adverse to the publisher’s commercial interest, and the organisation with the most complete knowledge of how the text was generated reached the conclusion that it could not reliably detect it. Every commercial claim to the contrary should be read against that.

Run Chapter 4’s arithmetic on it. Sensitivity 0.26, specificity 0.91. At a prevalence of one AI-written item in twenty, precision is about 13 per cent. At one in a hundred, about 2.8 per cent. Analysis The tool was not merely weak. Deployed at plausible rates it would have been wrong about roughly nineteen flags in twenty, and each wrong flag is an accusation against a named person.

Paraphrasing, and a limit that is not engineering

Suppose a detector were much better, and suppose the text carried a deliberate statistical watermark. Sadasivan and colleagues tested this with recursive paraphrasing. A watermarking scheme’s true-positive rate at a one per cent false-positive rate fell from 99 per cent to 15 per cent after five rounds of paraphrase, with AUROC falling from 99.8 to 67.9 per cent. Neural detectors, zero-shot classifiers and retrieval-based detectors were also broken, the last being specifically designed to resist paraphrase. C8.03 Established

The same work supplies something more fundamental than an attack. It connects the AUROC of the best possible detector to the total variation distance between the human and machine text distributions, and derives an impossibility result: as those distributions converge, even an optimal detector performs only marginally better than random.

This is a different kind of claim from anything in Part II. The detector problems in Chapters 4 to 7 were contingent: benchmarks were built badly, signals expired, channels removed evidence, adversaries perturbed inputs. Each could in principle be improved. Here the ceiling is set by how similar machine text is to human text, and the entire generative research programme is a sustained effort to make them more similar. Progress in generation is not an attack on text detection; it is the same quantity, measured from the other side.

Figure 8.2 — The impossibility result, drawn. A schematic of the paper’s theoretical framework, not plotted from data. Source: C8.03.

The question worth asking instead

For disinformation at scale, “was this text generated by a machine” is close to the least useful question available. A hostile influence operation is not characterised by the provenance of its sentences. It is characterised by being an operation: many accounts, one intent, coordinated in time, provisioned from a common source.

Coordination leaves traces that are not in the content at all. Account creation patterns, posting synchrony, shared infrastructure, identical engagement timing, the ordering in which a claim appears across venues. Paraphrasing removes the stylistic signal and leaves every one of those untouched, because they are properties of behaviour rather than of text.

	“Was this generated?”	“Is this coordinated?”
What is measured	Statistical typicality of style	Behaviour: timing, provisioning, synchrony
Defeated by	Paraphrase; or writing fluently	Operating slowly and separately, which costs the attacker scale
Who absorbs the false positives	Non-native speakers, 12× the rate	Activists, diaspora groups, legitimate campaigns
Feedback signal	None. See C7.03	Partial: an operation taken down is observably gone
Theoretical ceiling	Falls as generation improves	Not tied to generation quality
Actionable	Rarely. Provenance alone implies no remedy	Yes. Coordinated inauthentic behaviour is an enforceable category

Table 8.1 — The right-hand column is better on five rows and not clean on any of them. Row three is the reason it must not be sold as a solution. Source: author’s analysis, C8.04.

Do not read this as an endorsement of coordination analysis

The redirection is sound and the tooling is not innocent. Coordinated-behaviour detection has its own disparate impact: diaspora communities that post the same news because it concerns them, activist networks that publish in synchrony because that is what organising looks like, and minority-language communities with small, tightly connected media ecosystems all generate the signatures these systems reward.

The argument of this section is that coordination is the right question, not that current answers to it are safe. Both questions produce accusations against real people, and only one of them has a theoretical ceiling. Analysis

Practical scenario

Illustrative scenario · not a documented incident

Your organisation runs an AI-text detector over applications for a training programme. Of 600 applicants, 340 are writing in their second language. The tool flags 38 applications; a manager proposes that flagged applicants be asked to explain themselves before progressing.

Estimate, using Figure 8.1, roughly how many of those 38 flags you should expect to be second-language writers who wrote their own application. Then consider what the explanation request does to an applicant who did nothing wrong and cannot prove it, and whether your organisation could defend the process if asked to.

Executive implications

Do not use AI-text detection in any decision affecting a person. Not admissions, hiring, performance, discipline, visas, benefits or procurement. A documented twelve-fold disparity against non-native speakers makes this a discrimination mechanism with a technical finish, and the disparity is foreseeable rather than accidental.
If such a tool is already in use, the exposure is live. The evidence is published, the mechanism is understood, and foreseeability is the pivot on which Chapter 11 turns. Withdraw it before you are asked to justify it.
Never ask someone to prove they wrote something. It is not provable, the request is a sanction in itself, and it falls hardest on those least able to contest it.
Redirect disinformation spending from content to coordination. Paraphrase defeats the first and not the second, and only the second produces an enforceable finding.
For your own text, mark provenance rather than detect its absence. Declaring what you produced is tractable; proving what someone else produced is not. Chapter 10 takes this up.

Common misconceptions

Misconception 1 · “A detector reporting 95 per cent confidence is telling me about provenance”

It is reporting how statistically unsurprising the text is. The clearest proof is that enriching a human-written essay’s vocabulary with an LLM cut the false-positive rate from 61.3 to 11.6 per cent. More machine involvement, fewer machine accusations. The number is about fluency, and it is being read as authorship.

Misconception 2 · “Watermarking will settle this”

A watermarked scheme’s true-positive rate fell from 99 to 15 per cent at a one per cent false-positive rate after five rounds of paraphrase. Watermarking is genuinely valuable for marking your own output, which is Chapter 10’s subject. It does not survive an uncooperative intermediary, and the actors you worry about are uncooperative by definition.

Misconception 3 · “The false-positive rate will improve”

On this task it is structurally bounded from the wrong side. Detection depends on the distance between human and machine text distributions, and reducing that distance is the objective of the entire generative research programme. The ceiling falls as a by-product of ordinary progress, exactly as detector obsolescence did in Chapter 6, but here it applies to the best possible detector rather than to current ones.

Limitations and open questions

The bias figures are from seven detectors in 2023. Commercial tools have changed and vendors claim improvement. The mechanism argument, that perplexity-like measures penalise narrower expression, is what carries forward; specific rates should be re-measured rather than assumed.
The impossibility result is theoretical and carries assumptions. It bounds the best possible detector under its framework. It is not a claim that no text is ever detectable, and short, heavily templated machine output can remain distinguishable.
Coordination detection is asserted as better-posed, not as validated. No figures are offered for its accuracy, and the box above states its disparate-impact problem. Treating it as the answer would repeat the error this chapter documents.
Multimodal coordinated content is under-measured. Operations now pair generated text with generated imagery and voice, and there is no established evaluation for detecting the combination as opposed to each stream separately.
No current commercial accuracy claim is assessed here. As throughout the book, vendor figures without a published method are excluded rather than argued with.

Teaching materials

Discussion questions

If enriching an essay’s vocabulary with an LLM reduces AI-detection flags, what exactly is being detected, and what word should we use for it instead of detection?
OpenAI withdrew its own classifier. Why has that not ended the commercial market for text detection?
Is a tool with a twelve-fold disparate error rate ever acceptable if its aggregate accuracy is high?
Coordination analysis has its own false-positive victims. How would you set the threshold, and who should decide it?
Does the impossibility result mean institutions should stop asking whether text was generated? What should they ask instead?

Assessment questions

Compute precision for the withdrawn OpenAI classifier at two prevalences and interpret the result for a named individual. (10 marks)
Explain why text lacks a generator artefact and what detectors substitute, then derive the disparate-impact result from first principles. (20 marks)
Contrast the generation question and the coordination question across the rows of Table 8.1, and state where the latter is also unsafe. (15 marks)

Practical exercise

Find out whether any part of your organisation currently runs AI-text detection on material submitted by people. Most organisations contain at least one such tool that no committee ever approved.

Research assignment

Take any commercial AI-text detector and locate its published false-positive rate broken down by writer first language. Report what you find, including finding nothing.

Seminar topic

“Deploying AI-text detection on work submitted by people is indefensible on the published evidence.” Argue either side.

Chapter synthesis

Part II established that detection is a narrow instrument badly understood. Text is worse, and worse in a different way. The problems are no longer contingent. A detector on text cannot measure provenance because there is no trace of provenance to measure, so it measures statistical typicality instead and converts a question about machines into a judgement about how ordinary someone’s writing is. The consequence has been measured: 61.22 per cent false positives for non-native English writers against roughly five per cent for native writers, 97.8 per cent of human-written essays flagged by at least one of seven tools, and a false-positive rate that falls when an LLM is allowed to improve the vocabulary.

Above that sits a ceiling that engineering does not lift. Detection quality is bounded by the distance between the human and machine distributions, and closing that distance is the purpose of the field producing the generators. Chapter 6 showed detectors expiring as a side effect of quality work; here the same dynamic applies to the best detector that could ever exist.

The institutional conclusion is unusually blunt for this book, because the evidence is unusually clear. Text detection should not be used in decisions about people, and organisations currently using it are running a documented disparate-impact mechanism whose harm is foreseeable from published literature. The useful question is not whether a sentence was generated but whether a set of accounts is an operation, which paraphrase cannot defeat and which at least yields an enforceable finding. That question also harms people when answered badly, and saying so is part of answering it honestly.

Against Figure 0.1, Layer 2 has now failed on text for reasons of principle rather than practice, and Layer 3, platform and distribution, is where the coordination question lives. Chapter 9 turns to the threats still arriving: real-time cloning, text-to-video, agentic impersonation and synthetic identities at scale.