Where Detectors Break: The Domain Map

The pooled-calibration trap

At a pooled 5% false-positive setting, the same AI detector flags 0.96% of human-written news articles but 15.4% of human-written poems — a 16× spread. Per-domain false-positive rates, measured on 13,371 human documents from the RAID benchmark with cluster-bootstrapped confidence intervals, show that “the detector’s false-positive rate” does not exist. Each detector has its own breaking domain, and a single pooled threshold hides all of it.

When a vendor tells you their detector is calibrated to a 5% false-positive rate, they mean one number computed over one pooled pile of human text. That number is an average. Averages are where variance goes to die.

Here is what the average conceals. We scored 480,000 AI generations and 13,371 human documents from RAID — the largest public detector benchmark — with two open detectors, calibrated each to a pooled 5% false-positive rate, and then measured the false-positive rate per domain. The pooled setting is honored on average. It is honored almost nowhere in particular.

The false-positive map

Figure 1. Per-domain false-positive rates at a pooled 5% calibration. Each dot is the measured rate on human documents only; whiskers are 95% cluster-bootstrapped confidence intervals (500 resamples, seed 42). The vertical line marks the 5% the pooled calibration promised.

Domain	gpt2-base FP rate	95% CI	radar FP rate	95% CI
poetry	15.4%	13.7–17.1	8.6%	7.3–9.9
reviews	3.1%	2.0–4.1	20.4%	17.8–22.9
reddit	3.7%	2.9–4.5	10.7%	9.3–12.0
recipes	7.3%	6.1–8.5	0.1%	0.0–0.3
books	4.0%	3.1–4.9	2.0%	1.4–2.6
abstracts	1.6%	1.0–2.2	4.2%	3.1–5.2
wiki	3.2%	2.3–4.0	1.2%	0.7–1.7
news	0.96%	0.5–1.4	0.2%	0.0–0.4

Read the poetry row first. One human poem in six is flagged as machine-generated by a detector that was calibrated, in good faith, to flag one document in twenty. A poetry journal using this tool at the vendor-recommended setting would falsely accuse contributors at sixteen times the advertised rate.

Then read the reviews row — in the other direction. The second detector, a robustified Transformer that is otherwise far stronger, falsely flags 20.4% of human reviews: the worst cell in the entire map, on the domain where false accusations carry the most commercial weight. Its spread runs from 0.1% on recipes to 20.4% on reviews — a 185× range under one “5%” calibration.

Neither detector is broken. Both are doing exactly what a pooled threshold tells them to do. The threshold is the fiction.

Why each detector breaks somewhere different

The two detectors fail on different domains because they read different signals. The first is a shallow perplexity scorer: it asks how surprising each token is under a language model. Poetry, with its deliberate line breaks, unusual word choices, and compressed syntax, is surprising almost by definition — so it scores like machine text. The second is a fine-tuned Transformer classifier, and it learned its decision boundary from training data in which reviews are heavily machine-contaminated; it absorbed the genre’s conventions as an AI tell.

This is the mechanism under the map: a false-positive rate is not a property of a detector. It is a property of a detector crossed with a domain. Any procurement process that treats it as the former is buying a number that will not survive contact with a specific corpus.

One honest caveat on the map itself: the reviews domain contributes 943 human documents to RAID where other domains contribute roughly 1,780, so its confidence intervals are wider (±2.5pp on radar’s 20.4%). The effect dwarfs the uncertainty, but the sample-size asymmetry is part of the record.

Domains under attack

The false-positive map is about human text. The other half of the domain story is what happens to machine text in each domain when an adversary perturbs it. We measured detection accuracy per domain under each of RAID’s eleven attacks; the worst cells are all the same attack, and the spread across domains is enormous.

Domain	gpt2-base clean	gpt2-base + homoglyph	radar clean	radar + homoglyph
books	74.3%	0.4%	91.9%	13.3%
news	73.6%	0.8%	80.7%	17.5%
reviews	79.3%	4.8%	94.5%	23.5%
recipes	80.8%	7.2%	86.4%	25.0%
wiki	64.7%	4.6%	77.7%	6.7%
reddit	69.3%	2.4%	82.2%	11.2%
poetry	55.6%	3.9%	87.2%	22.7%
abstracts	41.0%	0.8%	80.2%	7.8%

Table 2. Detection accuracy on machine-generated text, clean vs. homoglyph-attacked, per domain. Clean rates are per-domain generation catch rates; attacked rates use per-row pairing. Confidence intervals on domain×attack cells run ±1.4pp (gpt2-base) and ±3.1pp (radar).

Two findings live in this table. First, even clean detection accuracy is a domain variable: the shallow detector catches 80.8% of machine-written recipes but only 41.0% of machine-written abstracts. Second, the homoglyph attack — swapping Latin characters for lookalike Unicode — collapses every domain for both detectors, but from wildly different starting points. A detector that looks acceptable on its home domain (books: 91.9% clean) is a coin flip’s worse half after one find-and-replace.

The per-document version of this collapse — 93.7% of attacked documents literally flip their verdict — is the subject of our companion piece on the flip rate. The domain map is where that collapse lands: not evenly, and not where you would guess.

The generator skew

There is one more axis on which “the detector’s accuracy” dissolves: which model wrote the text.

Figure 2. Clean detection accuracy by generator, per detector, with 95% cluster-bootstrapped CIs. Each detector has its own easiest and hardest generator; the spans are 43.9pp (gpt2-base) and 24.2pp (radar).

Generator	gpt2-base	95% CI	radar	95% CI
gpt4	40.0%	39.2–40.9	74.1%	71.9–76.4
chatgpt	60.9%	59.9–62.1	81.2%	78.9–83.4
mistral	68.9%	68.2–69.7	57.0%	54.8–59.3
llama-chat	83.9%	83.0–84.8	79.8%	77.7–81.9

The shallow detector catches 83.9% of llama-chat text but only 40.0% of gpt4 text — worse than a coin flip on the generator your content farm is most likely to pay for. The robustified detector is tighter but still spans 24 points, and its weakest generator (mistral, 57.0%) is a different model from the shallow detector’s weakest (gpt4, 40.0%). A vendor number quoted on “GPT-4 text” and a vendor number quoted on “AI text” are not the same claim, and neither transfers.

What Stanford’s 61.3% can and can’t tell you

The most-cited number in this space is from Liang et al.’s 2023 study: GPT detectors misclassified 61.3% of TOEFL essays written by non-native English speakers, while near-perfectly scoring native speakers’ essays. It is the right study to cite on detector bias, and it is worth being precise about what our data adds — and what it cannot add.

RAID’s human documents carry no writer-language labels. We cannot reproduce, extend, or refute the non-native-writer finding, and nothing in this article should be read as doing any of those. What we can do is show that the same structural problem — one pooled number standing in for a distribution — exists along the domain axis and the generator axis, measured at a scale and with confidence intervals the earlier work did not attempt. The bias finding and the domain map are the same warning in two dialects: a detector’s error rate is a distribution, and the tail you care about is the one your corpus lives in.

What to do with this

If you evaluate, buy, or enforce AI detection, three practices follow directly from the map:

1. Demand per-domain false-positive rates, not pooled ones. A vendor who cannot produce an FP rate on your corpus genre — poetry, reviews, abstracts, whatever yours is — has not answered the question that matters. The pooled number they gave you is compatible with a 16× spread.
2. Calibrate on your own corpus before enforcing anything. Run the detector on a few hundred known-human documents from your actual domain. If the measured FP rate exceeds your tolerance, no vendor datasheet outranks your own measurement.
3. Treat a single flagged document as a signal, not a verdict. In the worst cells of this map, one human document in five is flagged. An accusation built on one detector score is an accusation built on the domain lottery.

The deeper alternative is to stop depending on detection at all. Detectors estimate what text looks like; provenance records what text is. That is the premise of Violet’s Signal Integrity Protocol — the system this research program exists to stress-test.

FAQ

What is a false-positive rate in AI detection? The share of human-written documents a detector incorrectly flags as AI-generated. At a 5% false-positive rate, one human document in twenty is falsely accused. This article shows the rate is not one number: measured per domain, it ranged from 0.96% (news) to 15.4% (poetry) for one detector and 0.1% (recipes) to 20.4% (reviews) for another, under the same pooled 5% calibration.

Which writing domains trigger the most false positives? It depends on the detector. The shallow perplexity-based detector we tested spiked on poetry (15.4% FP); the robustified Transformer spiked on reviews (20.4% FP). News and recipes were the safest domains respectively. There is no universally “safe” or “dangerous” genre — only detector-and-domain pairs.

Is a 5% FPR pooled calibration trustworthy? Not for enforcement decisions on a specific genre. Pooled calibration guarantees the average across a mixed corpus; per-domain rates deviated from it by up to 15 points in our measurements. Trust per-domain measurements on your own corpus instead.

Do detectors behave the same across AI models? No. Clean detection accuracy spanned 40.0% to 83.9% across four generator families for one detector and 57.0% to 81.2% for the other. Each detector also had a different weakest generator, so a score quoted on one model’s output does not transfer to another’s.

---

Method note: all numbers are from Violet’s reanalysis of the RAID benchmark (Dugan et al., ACL 2024, code), 5,000-source sample (seed 42): 480,000 generations across 4 model families × 2 decoding settings, plus 13,371 clean human documents. Detectors: gpt2-base (perplexity scorer) and radar (robustified Transformer), each calibrated to a pooled 5% false-positive rate on the pooled human set. Confidence intervals are 95% cluster-bootstrapped by source document (500 resamples, seed 42). Human rows in RAID are clean-only, so attacked-human false-positive cuts do not exist; two further detectors (Binoculars, GLTR) were excluded for compute feasibility, with partial results preserved. All other deviations from the benchmark’s published protocol are confined to sampling scope: four of eleven model families, and no-penalty decoding rows. Full data and methods: the RAID reanalysis data page.