AI Detector Disagreement: Every Published Number (2026)
Run one essay through three AI detectors and you can get three different answers. That is not a rumour or a forum anecdote — it is a measured, published property of the tools, and the size of the gap has now been quantified in peer-reviewed and conference research. This page collects every credible published number on how much AI detectors disagree with each other, drawn only from academic papers and university research offices. No vendor marketing claims are used as a source for any figure here.
Want to bypass Turnitin in 2026? Grab the free prompt pack.
Get the exact text-humanization prompts I use to drop an AI score by hand — copy, paste, submit. Free, straight to your inbox.
Send me the free prompts →Five leading commercial AI detectors were run over the same corpus of roughly 6,000 papers and their AI-generated clones. Their false positive rates ranged from 0.05% to 68.6% — the same texts, the same test, and more than a 1,300-fold difference in how often a human author gets flagged.
Source: Layton, Madeiros, Butler & Traynor, 2026 IEEE Symposium on Security and Privacy (University of Florida, May 2026)
Key Takeaways
- 0.05%–68.6% — false positive range across five commercial detectors on one shared corpus (IEEE S&P, 2026)
- 0.3%–99.6% — false negative range across those same five detectors (IEEE S&P, 2026)
- 0 of 120 — detector pairs among 16 systems whose scores correlated above 0.8; only 5.5% cleared 0.6 (Stowe et al., 2026)
- 6 of 16 — detection systems scoring at or below a coin flip on a balanced benchmark (Stowe et al., 2026)
- 0%–50% — false-accusation risk across 14 tools tested on identical documents (Weber-Wulff et al., 2023)
- 9/9 vs 0/9 — verdicts four tools and Content at Scale returned on the very same nine ChatGPT-written documents (Weber-Wulff et al., 2023)
- 0.449–0.526 — accuracy of expert human annotators asked to do the same job, close to chance (Stowe et al., 2026)
On this page
Want to bypass Turnitin in 2026? Grab the free prompt pack.
Get the exact text-humanization prompts I use to drop an AI score by hand — copy, paste, submit. Free, straight to your inbox.
Send me the free prompts →1How far apart are the error rates?
Far enough that “the detector said so” carries almost no information on its own. The most recent hard measurement comes from a University of Florida team presenting at the 2026 IEEE Symposium on Security and Privacy. They took roughly 6,000 papers submitted to top-tier security conferences before ChatGPT existed — so the human authorship is certain — then had large language models produce AI clones of those same papers. Five of the most popular commercial detectors were run across the combined set.
| Metric | Value | Source |
|---|---|---|
| False positive rate range, 5 commercial detectors | 0.05% – 68.6% | IEEE S&P 2026 / Univ. of Florida |
| False negative rate range, same 5 detectors | 0.3% – 99.6% | IEEE S&P 2026 / Univ. of Florida |
| Detectors that performed well before adversarial testing | 2 of 5 | IEEE S&P 2026 / Univ. of Florida |
| Corpus size | ~6,000 papers | IEEE S&P 2026 / Univ. of Florida |
The researchers then applied what they call a lexical complexity attack: they simply asked the model to rewrite the AI version using more sophisticated vocabulary. That alone was enough to render the detectors, in the team’s words, basically useless at telling AI text from human text. This is the same brittleness that shows up in published bypass rates after humanization — a small change in surface style moves the score far more than any change in who actually wrote the words.
Lead author Patrick Traynor’s summary is worth quoting directly, because it is the plainest statement of the problem from someone who measured it: “These are not reliable or robust tools to use to measure the problem. We really can’t use them to adjudicate these decisions. People’s careers are on the line here.”
2Do any two detectors actually agree?
Statistically, almost none of them do. A 2026 study of 16 detection systems computed the correlation between every pair of detectors scoring the same texts. The result is the single most direct measurement of disagreement in the literature.
With 16 systems there are 120 possible pairings. None of them reached the 0.8 threshold that would suggest two tools are broadly measuring the same thing. That is the statistical shape of the experience students describe when one detector says human and Turnitin says AI: the tools are not noisy versions of a single underlying verdict, they are substantially different instruments.
Raw capability varies just as widely. On a balanced benchmark of human and machine text, AUROC scores across the 16 systems spanned an enormous range.
An AUROC of 0.5 is a coin flip. Six of the 16 systems scored at or below 0.477, meaning they carried essentially no diagnostic value on this benchmark — and one family of tools, BiScope, ranged from 0.362 to 0.805 depending purely on which dataset its variant had been trained on. Same underlying method, wildly different behaviour. This is a large part of why Turnitin flags AI when other detectors don’t: the tools were built on different training data, weight different signals, and set their thresholds independently.
3The same nine documents, fourteen verdicts
The clearest demonstration of disagreement remains a 2023 study in the International Journal for Educational Integrity, which ran 14 detection tools over an identical set of 54 documents — 756 individual tests in total. Because every tool saw exactly the same documents, any difference between tools is pure disagreement rather than a difference in test material.
Same documents, different verdicts
Pick a document type. Each bar shows how many of those nine identical documents each tool classified correctly.
Set the selector to unedited ChatGPT text and the spread speaks for itself: Check For AI, Crossplag, Go Winston and the GPT-2 Output Detector each caught all nine documents, while Content at Scale caught none of them. Switch to machine-translated human writing and the ranking inverts — Compilatio and Turnitin classify all nine correctly while Check For AI gets none right. No single tool leads across all six document types, which is precisely why a score from one detector cannot be treated as a verdict. If you are trying to interpret a number you have been handed, our guide to what AI detection scores really mean covers how to read it in context.
| Metric | Value | Source |
|---|---|---|
| Tools tested on identical documents | 14 tools × 54 docs = 756 tests | Weber-Wulff et al., IJEI 19:26 |
| Accuracy spread across tools (binary method) | 76% – 33% | Weber-Wulff et al., Table 7 |
| False-accusation (false positive) rate range | 0% – 50% | Weber-Wulff et al., Table 13 |
| False-negative rate range | 8.3% – 100% | Weber-Wulff et al., Table 14 |
| Tools averaging 80%+ accuracy | 0 of 14 | Weber-Wulff et al., discussion |
| Tools exceeding 70% accuracy | 5 of 14 | Weber-Wulff et al., discussion |
The false-positive range is the number with real-world consequences attached. Seven of the fourteen tools never once misclassified a human document; one tool, GPT Zero, misclassified half of them. A student’s exposure to a false accusation therefore depended almost entirely on which product their institution had bought — a procurement decision, not an evidentiary one. That asymmetry is part of why a growing list of universities have switched their AI detectors off entirely.
4Detectors also disagree with themselves
Cross-tool variance is only half the picture. The same study found single tools returning materially different results on document sets built by an identical process.
The authors’ own caution is the operative point: they noted results can vary when the same material is tested at a different time, and concluded this instability “can have major implications in misconduct investigations and thus provides another strong reason against the use of these tools as a single source of an accusation of misconduct.” If a score can move without the text moving, a screenshot of that score is not evidence of anything — which is why version history and drafting records carry far more weight in an appeal than a detector percentage does.
5Disagreement widens on edited and paraphrased text
Detectors are most consistent on the case that matters least — clean, untouched human writing — and least consistent on the cases that actually reach a misconduct panel. Averaged across all 14 tools, accuracy collapses as text is edited.
Average accuracy falls from 94% on human writing to 30% on AI text a human has edited, and to 15% on AI text a paraphrasing tool has rewritten. The corresponding average false-negative rate on paraphrased text reaches 71.4%. Note what this means for the two most common real-world scenarios: a student who drafted with AI and then rewrote it themselves, and a student who ran AI output through a rewriting tool, both land in the bands where the tools are least reliable and least consistent with each other.
6Who absorbs the disagreement
Detector variance is not distributed evenly across writers. Two findings, six years and two research teams apart, point the same direction.
The 2026 study found the same pattern in a different population and at greater depth. Across 41,743 student essays, essays by English-language learners were more likely to be classified as machine-generated, and subgroup analysis showed non-White ELL essays disproportionately flagged by seven models compared with just one model for their White counterparts. This is the mechanism behind the documented ESL bias in AI detection — not a single biased tool, but a set of tools whose disagreements happen to cluster on the same group of writers.
One counter-intuitive result is worth keeping: the researchers also measured a negative correlation between a detector’s accuracy and its estimated bias (r = −0.486), meaning better-performing detectors tended to be somewhat fairer. It is a trend rather than a guarantee, and it does not license trusting any individual tool without testing it.
The same study handed the task to three expert human annotators, who scored between 0.449 and 0.526 accuracy — barely distinguishable from guessing. Humans were worse than the good detectors at accuracy, but showed no statistically significant demographic bias. That contrast is the practical argument for human review as a check rather than a replacement, and it is why institutional policy increasingly turns on whether a detector score can be used as proof at all. If you are facing a flag right now, the immediate priorities are set out in our first-24-hours guide.
Methodology and limitations
Research date: 22 July 2026. Sources consulted: 9. Sources cited: 5. Every figure on this page comes from a peer-reviewed journal article, an arXiv preprint, or a university research office reporting its own study. Vendor blogs, detector marketing pages and humanizer-company content were excluded as sources for all numbers.
Freshness distribution: three sources are from 2026, two from 2023. The 2023 figures are retained because no superseding study has re-run those specific comparisons, and they are labelled with their year wherever they appear. Readers should note the 2023 tests used the ChatGPT model of 13 February 2023 and therefore describe GPT-3.5-era detection, not current-generation models.
Disclosures. The 16-system study (Stowe et al.) was produced by researchers employed at Pindrop, a commercial detection vendor; its authors state the limitation that their findings should not be assumed to generalise across all systems, and that their corpus is entirely US student essay writing. In the 2023 14-tool study, two authors disclosed that a conference they organise receives sponsorship from Turnitin and Compilatio — both of which rank highly in that paper’s results. The ACL 2026 presentation venue for the 16-system study is stated by Pindrop’s own write-up and does not appear in the arXiv preprint.
Known limits. The 120-pair figure in Section 2 is simple arithmetic from the 16 systems tested, not a count printed in the paper. No study located here publishes a document-by-document table of every tool’s verdict on every individual text; the per-document-class figures used in the interactive tool are the finest-grained public data available. Update schedule: reviewed quarterly, or whenever a new multi-detector comparison is published.
Frequently asked questions
How much do AI detectors actually disagree with each other?
Substantially. In a 2026 study of 16 detection systems, no pair of detectors correlated above 0.8 and only 5.5% of pairs exceeded 0.6. A separate 2026 IEEE Symposium on Security and Privacy paper ran five commercial detectors over one shared corpus and found false positive rates spanning 0.05% to 68.6%.
Can two AI detectors give opposite verdicts on the same essay?
Yes, and it is documented. On the same nine ChatGPT-written documents, four tools flagged all nine while Content at Scale flagged none. On the same nine machine-translated human texts, Compilatio and Turnitin classified all nine correctly while Check For AI got none right.
Does the same detector always give the same score on the same text?
No. Researchers testing two document sets prepared by an identical method saw DetectGPT score 2/9 and then 8/9. When they re-uploaded documents to check, they obtained different values again and confirmed it was tool inconsistency rather than testing error.
Which detector is most likely to falsely accuse a human writer?
It varies enormously by tool. In the 14-tool study, false-accusation risk ranged from 0% for seven tools to 50% for GPT Zero. Risk concentrated sharply on machine-translated writing: 14 of the 17 total false-positive events involved text a human wrote in another language and then translated. See our breakdown of AI detection false positive rates for per-tool figures.
Are humans better than detectors at spotting AI writing?
Not at accuracy. Three expert annotators scored between 0.449 and 0.526 — roughly a coin flip. The notable finding is that the humans showed no statistically significant demographic bias, while several detectors did, which is the main argument for keeping human review in the loop rather than replacing it.
Do detectors still disagree on edited or paraphrased AI text?
The disagreement widens. Average accuracy across 14 tools fell from 94% on human-written text to 30% on manually edited AI text and 15% on machine-paraphrased AI text, with the average false-negative rate on paraphrased text reaching 71.4%.
Does a higher-accuracy detector mean a fairer detector?
It correlates, but weakly enough to guarantee nothing. Researchers found a negative correlation between accuracy and estimated bias (r = −0.486). In the same study, non-White English-language-learner essays were disproportionately flagged by seven models versus one for their White counterparts. Our page on AI detection false negative rates covers the opposite failure mode.
Sources
- University of Florida News. “Watching the detectors: Researchers probe efficacy – and danger – of AI detection tools.” news.ufl.edu. Reporting Layton, Madeiros, Butler & Traynor, “AI Wrote My Paper and All I Got Was This False Negative: Measuring the Efficacy of Commercial AI Text Detectors,” 2026 IEEE Symposium on Security and Privacy. Accessed 22 July 2026.
- Stowe, K., Afanaseva, S., Raimundo, R., Sun, Y. & Patil, K. “Identifying Bias in Machine-generated Text Detection.” arXiv:2512.09292v2 [cs.CL], 23 April 2026. arxiv.org/abs/2512.09292. Accessed 22 July 2026.
- Pindrop. “Is AI Text Detection Biased? What Testing 16 Systems Revealed.” 24 June 2026. pindrop.com. Accessed 22 July 2026.
- Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P. & Waddington, L. “Testing of detection tools for AI-generated text.” International Journal for Educational Integrity, 19:26 (2023). doi.org/10.1007/s40979-023-00146-z. Accessed 22 July 2026.
- Elkhatat, A. M., Elsaid, K. & Almeer, S. “Evaluating the efficacy of AI content detection tools in differentiating between human and AI-generated text.” International Journal for Educational Integrity, 19:17 (2023). doi.org/10.1007/s40979-023-00140-5. Accessed 22 July 2026.
Last updated: 22 July 2026 · Detection Drama Research Desk
