AI Detector Robustness: Data and Methods from Our RAID Reanalysis
Full aggregate tables from Violet's RAID reanalysis: detector accuracy under 11 attacks, per-document flip rates, per-domain false positives, methods.
This page is the canonical record of the measurements behind Violet’s AI-detection research cluster. Every number quoted in The Flip Rate, Where Detectors Break, Can a Paraphraser Beat AI Detection?, and You Cannot De-Watermark Prose traces to a table on this page.
Two open detectors, calibrated to a pooled 5% false-positive rate, were scored on 480,000 machine generations and 13,371 human documents from the RAID benchmark (Dugan et al., ACL 2024). Under the single most damaging attack, accuracy falls from 63.4% to 4.2% — and 93.73% of individual documents change verdict.
Sample and scope
- Sources: 5,000 documents sampled from the RAID training split (seed 42).
- Generations: 480,000 — four model families (ChatGPT, GPT-4, Llama-Chat, Mistral) × two decoding settings, restricted to rows with no repetition penalty.
- Human documents: 13,371, clean only (RAID contains no attacked-human rows), used for threshold calibration.
- Detectors: gpt2-base (a shallow perplexity scorer) and radar (a robustified Transformer classifier).
- Attacks: the eleven transformations defined by the RAID benchmark, including paraphrase via DIPPER.
Scope deviations from the paper are confined to the model set (four of eleven families) and decoding (no-penalty rows only).
Detector decay matrix
Aggregate accuracy per detector per attack, with 95% cluster-bootstrapped confidence intervals (500 resamples, seed 42). Machine-generated rows; a correct answer is “machine.”
| Attack | gpt2-base accuracy (95% CI) | radar accuracy (95% CI) |
|---|---|---|
| Clean (no attack) | 63.4% (62.9–64.1) | 73.0% (71.3–74.6) |
| Homoglyph | 4.2% (4.0–4.4) | 22.4% (21.1–23.5) |
| Upper/lower case | 10.6% (10.3–10.8) | 71.1% (69.6–72.9) |
| Article deletion | 30.5% (29.9–31.1) | 66.6% (64.9–68.4) |
| Targeted misspelling | 44.4% (43.7–44.9) | 71.5% (70.0–73.1) |
| Whitespace | 49.2% (48.6–49.9) | 63.7% (61.9–65.4) |
| Alternative spelling | 58.3% (57.7–59.0) | 73.2% (71.6–74.7) |
| Number swap | 59.9% (59.3–60.5) | 73.8% (72.4–75.3) |
| Insert paragraphs | 61.0% (60.4–61.7) | 82.1% (81.0–83.3) |
| Paraphrase (DIPPER) | 68.8% (68.3–69.5) | 56.0% (54.5–57.7) |
| Synonym shuffle | 72.5% (71.9–73.2) | 67.1% (65.3–68.9) |
| Zero-width space | 100.0% (99.9–100.0) | 98.9% (98.6–99.1) |
Table 1. The zero-width-space row is an inversion, not robustness: the attack destroys the textual signal in a direction that happens to raise measured accuracy. Interpretation in You Cannot De-Watermark Prose.
Per-document flip rates
The fraction of documents whose verdict changes when the same document passes through one attack — the metric aggregate accuracy hides. Full argument in The Flip Rate.
| Attack | gpt2-base flip rate | radar flip rate |
|---|---|---|
| Homoglyph | 93.73% (CI 93.4–94.0) | 78.69% (CI 77.3–80.1) |
| Upper/lower casing | 83.67% (CI 83.3–84.1) | 5.70% (CI 5.0–6.4) |
| Article deletion | 52.12% (CI 51.4–52.8) | 8.94% (CI 8.0–9.8) |
| Whitespace | 23.05% (CI 22.5–23.6) | 14.64% (CI 13.6–15.7) |
| Paraphrase | 16.85% (CI 16.3–17.4) | 33.80% (CI 32.1–35.6) |
| Alternative spelling | 8.26% (CI 7.9–8.6) | 0.72% (CI 0.5–0.9) |
| Number substitution | 5.67% (CI 5.3–6.1) | 1.52% (CI 1.2–1.9) |
| Insert paragraphs | 5.54% (CI 5.3–5.8) | 2.88% (CI 2.4–3.4) |
| Synonym | 4.72% (CI 4.4–5.0) | 10.14% (CI 9.3–11.1) |
| Zero width space | 0.03% (CI 0.01–0.05) | 1.15% (CI 0.9–1.4) |
Table 2. Attacked rows are paired to their exact clean siblings per row via RAID’s adv_source_id field. Machine-generated rows only; RAID contains no attacked-human rows, so false-positive flips do not exist in this data.
Per-domain false-positive rates
Measured on the 13,371 human documents at the same pooled 5% calibration. Each cell is the rate at which human writing in that domain is flagged as machine. Full analysis in Where Detectors Break.
| Domain | gpt2-base FP rate | 95% CI | radar FP rate | 95% CI |
|---|---|---|---|---|
| poetry | 15.4% | 13.7–17.1 | 8.6% | 7.3–9.9 |
| reviews | 3.1% | 2.0–4.1 | 20.4% | 17.8–22.9 |
| 3.7% | 2.9–4.5 | 10.7% | 9.3–12.0 | |
| recipes | 7.3% | 6.1–8.5 | 0.1% | 0.0–0.3 |
| books | 4.0% | 3.1–4.9 | 2.0% | 1.4–2.6 |
| abstracts | 1.6% | 1.0–2.2 | 4.2% | 3.1–5.2 |
| wiki | 3.2% | 2.3–4.0 | 1.2% | 0.7–1.7 |
| news | 0.96% | 0.5–1.4 | 0.2% | 0.0–0.4 |
Table 3. The reviews domain contributes 943 human documents where other domains contribute roughly 1,780, so its intervals are wider; the effect dwarfs the uncertainty.
Per-generator detection accuracy
Clean detection accuracy by generator family. A vendor number quoted on one generator does not transfer to another.
| Generator | gpt2-base | 95% CI | radar | 95% CI |
|---|---|---|---|---|
| gpt4 | 40.0% | 39.2–40.9 | 74.1% | 71.9–76.4 |
| chatgpt | 60.9% | 59.9–62.1 | 81.2% | 78.9–83.4 |
| mistral | 68.9% | 68.2–69.7 | 57.0% | 54.8–59.3 |
| llama-chat | 83.9% | 83.0–84.8 | 79.8% | 77.7–81.9 |
Table 4. Spans: 43.9 points (gpt2-base) and 24.2 points (radar), with different weakest generators per detector.
Paraphrase divergence
The paraphrase attack deserves a separate view because the two detector families move in opposite directions. Full analysis in Can a Paraphraser Beat AI Detection?.
| Detector | Clean accuracy | After paraphrase | Δ | 95% CI |
|---|---|---|---|---|
| gpt2-base (perplexity scorer) | 63.4% | 68.8% | +5.4pp | ±0.6 |
| radar (robustified Transformer) | 73.0% | 56.0% | −17.0pp | ±1.6 |
Table 5. 40,000 paraphrased documents via DIPPER. Evasion through paraphrasing is a detector lottery, not a technique.
Methods
Calibration. Each detector’s threshold was set on the pooled human set to a 5% false-positive rate — the vendor-recommended operating point style used by the benchmark.
Pairing. Attacked rows are paired to their clean siblings per row via the dataset’s adv_source_id field. An earlier per-source pairing collapsed eight model-decoding variants into one and produced an inflated 99.96% flip estimate; the corrected per-row pairing yields the 93.73% figure reported above. Both numbers and the fix are on the record.
Confidence intervals. A row-IID bootstrap treats 480,000 generations as independent draws, but the 96 rows per source share a prompt, a generator, and a decoding family. We therefore cluster the bootstrap at the source level (500 resamples, seed 42), which is the statistically defensible interval.
Validation. The pipeline was validated against the paper’s Table 6 within the documented scope deviations, with direction-of-effect agreement on every detector–attack pair.
Exclusions. Two further detectors from the original run set are excluded: Binoculars (runtime-infeasible after an Apple Silicon port, stopped at 80 rows) and GLTR (stopped at 1.6% — marginal third-column value). Partial outputs are preserved; the exclusions are stated, not dropped.
Cite this analysis
The aggregate tables on this page are published under CC BY 4.0. Suggested citation:
Korovkin, M. (2026). AI Detector Robustness: Data and Methods from Our RAID Reanalysis. Violet. https://hi-violet.com/research/raid-detector-reanalysis
@misc{violet-raid-reanalysis-2026,
author = {Korovkin, Michael},
title = {AI Detector Robustness: Data and Methods from Our RAID Reanalysis},
year = {2026},
publisher = {Violet},
url = {https://hi-violet.com/research/raid-detector-reanalysis},
note = {Aggregate measurements from a reanalysis of the RAID benchmark (Dugan et al., ACL 2024)}
}Frequently asked questions
How many AI generations were analyzed? 480,000 generations from 5,000 sampled sources (seed 42) across four model families and two decoding settings, plus 13,371 clean human documents used for threshold calibration.
Which AI detector is most robust to attacks? Of the two measured, radar (a robustified Transformer) holds up better overall but still falls to 22.4% accuracy under homoglyph attack and loses 17.0 points under paraphrase. Neither detector is robust in absolute terms.
Can I reuse this data? Yes. The aggregate tables on this page are published under CC BY 4.0; cite Violet’s RAID reanalysis data page. The underlying benchmark is RAID (Dugan et al., ACL 2024).
Bring the market problem into focus.
Turn social and market signals into an ecosystem-growth, go-to-market, or technical engagement built around the work your team needs.
Explore engagements