Substack’s AI Detector: How the Pangram Check Works, How Accurate It Is, and How to Turn It Off

By Vlad Ivanov · Runs Detection Drama and tests AI detectors and humanizers hands-on
Last updated: October 9, 2026 · 9 min read

Substack’s AI detector is a Pangram scan any reader can run on a post, Note or comment over 100 words. It is the most accurate commercial detector in published tests, but it returns a probability, not proof, and writers can switch it off one post at a time.

Within a week of launch, r/Substack had threads with 381 comments on the new tool, a running list of alleged errors and how to disable it. The questions underneath the anger are practical: is the score right, who can see it, and how do you turn it off. This guide answers those with the published evidence, and is clear about where that evidence stops.

What is Substack’s AI detector?

It is an in-app check, built with the detection company Pangram and announced by co-founder and CEO Chris Best on 21 July 2026, that estimates how much of a piece of writing was produced with AI. It sits alongside an optional “How I make this” statement where writers can describe their process, including any AI use.

Substack frames it as transparency rather than enforcement. A spokesperson told 404 Media the tools are “intended to increase transparency and give readers more context, not to prohibit or penalize AI-assisted writing.” Best himself wrote: “We’re not against people using AI to assist their work.”

If you want the background on the detector itself, our Pangram review covers how Pangram performs as a standalone product. This article is about the Substack integration.

How does the Substack AI check work?

A reader opens the menu on a post and asks for a scan; Pangram returns a percentage estimate of AI involvement, shown as an overlay on the content. Nothing is scanned until someone requests it. According to Engadget, the check works on the web and iOS first, with Android to follow, and it only runs on text longer than 100 words published from launch day onward.

Two details matter for writers. First, Pangram is designed to pick up AI used for editing as well as full generation, which is why SAN’s reporting says it can tell whether AI produced the text but not whether AI was used as a research tool. Second, Pangram says it trained on human text written before 2021, which is the basis for its low false positive claim.

How accurate is Pangram on Substack?

On published evidence, Pangram is the strongest commercial detector available, but the numbers come from controlled tests, not from Substack’s live feed. Pangram’s own figure is a 0.01% false positive rate for its 3.3 model; CEO Max Spero put it to 404 Media at roughly one in 10,000.

The independent number comes from the working paper Artificial Writing and Automated Detection by Brian Jabarian and Alex Imas at the University of Chicago. On a dataset of about 2,000 human-written passages across six kinds of writing, with AI versions from four large language models, Chicago Booth Review reports that Pangram’s false positive rate was “essentially 0 across most decision thresholds” and its false negative rate 2 to 4 percent. The authors called it “the only AI detector maintaining policy-grade levels on our main metrics.”

What the accuracy numbers do not tell you

A low false positive rate is a rate, not a guarantee. As a worked illustration using Pangram’s own one-in-10,000 estimate: if a million genuinely human pieces were scanned, about 100 would still come back looking like AI. On a platform where any reader can scan any eligible post, those people exist, and from the outside they look identical to writers who did use AI.

The tests also measure clean cases: human text versus raw model output. Most contested Substack posts sit in between, human drafts polished with a grammar tool or partly rewritten by a chatbot. Media Copilot notes that lightly edited or paraphrased AI text is “far harder to catch than raw model output,” and The Atlantic’s Matteo Wong, quoted by SAN, wrote that the tool “does make mistakes, perhaps to a greater extent than is currently understood.” Chris Best’s own line is that “It is not perfect, but independent research suggests that it detects AI-generated text with a high degree of accuracy.”

Non-native English writing is the other known weak spot for detectors as a category; our breakdown of AI detection accuracy by language and the GPTZero ESL false positive data show how much that varies by tool.

Why are human writers getting flagged on Substack?

The pattern in writer reports is edited text, not untouched prose. In one r/Substack thread, a writer described a post scored at 100 percent AI while other checkers said 0 percent; the top replies pointed to Grammarly’s AI rewrite suggestions. Those are user reports, not verified tests, but they match how Pangram is built: it looks for AI-assisted editing, not only AI drafting.

The same thing happens in classrooms, which we covered in Grammarly triggering Turnitin’s AI flag and whether detectors flag tutor-edited writing. Other writers on r/Substack report flags on essays written years before ChatGPT; we could not verify any of those cases from the threads, and one commenter complained that such reports rarely include the text. Treat both the complaints and the defences as anecdote.

How do you turn off the AI detector on Substack?

You disable it per post or per note; as of October 2026 there is no single setting for a whole publication. The steps below were documented by a Substack writer in July 2026 and match what writers describe on r/Substack. Substack is still changing the interface, so menu labels may shift.

1. Open the draft and click Continue (Next on mobile) to reach the Publishing page.
2. Click “Scan for AI text” (on mobile, tap the three dots and choose “Check for AI”).
3. Open the three-dot menu inside the analysis box once the report appears.
4. Select “Disable detection.” Readers will see “AI detection unavailable” instead of a score.
5. For Notes and comments, use the three dots on the note, choose “Check for AI”, then “Disable detection” in the result box.

What should you do if Substack flags your writing as AI?

1. Check the result is real. Only text over 100 words published after 21 July 2026 is eligible; a screenshot of an older post scored elsewhere is a different tool.
2. Re-scan the draft yourself and look at which passages drive the score. Grammar-tool rewrites and pasted summaries are the usual suspects.
3. Keep your process evidence. Version history in Google Docs or a writing process tracker is stronger proof of authorship than any detector score.
4. Add a “How I make this” statement describing your process honestly, including any AI help with research or editing.
5. Report or remove the result if you believe it is wrong, or disable detection on that post.

If you did use AI to draft, the honest options are disclosure or rewriting in your own voice; our guide on making ChatGPT drafts sound like you covers the second. We have also tested how individual rewriting tools fare against Pangram, for example Text Polish against Pangram, and our Pangram bypass breakdown explains why a single trick rarely holds.

Does an AI label hurt your reach on Substack?

Not algorithmically, according to Substack: a spokesperson told 404 Media the tools “do not impact discovery on the platform.” The cost writers fear is social. Ghostwriter Alice Lemee told the same outlet that “All it takes is one false accusation for a writer to have their reputation almost irreversibly tarnished.”

The base rate is also higher than many writers assume. In Pangram’s own report, drawn from more than a million posts scanned through its Chrome extension, Substack had the lowest combined AI rate of the longform platforms measured, yet more than a fifth of its posts were flagged as AI-generated or AI-assisted. Readers who scan will see a lot of non-zero scores. Similar platform-level enforcement numbers are in our Wikipedia AI enforcement data.

How does Substack’s checker compare with other AI detectors?

In the University of Chicago test it beat the two other commercial tools on the combination that matters for a public label, very few false accusations and few misses. All three kept false positives under 1 percent; the difference was in what they missed.

Source: Jabarian and Imas, University of Chicago, as reported by Chicago Booth Review.

Detectors also disagree with each other on the same text, which is why a single score from any of them should start a conversation rather than end one; see what to do when detectors disagree and our AI text watermarking statistics for where watermarking fits.

How we put this together

We read more than 25 r/Substack threads about the feature posted between July and October 2026, each with 38 to 381 comments, and the launch coverage from Engadget, 404 Media, SAN, Media Copilot and Net Influencer. Accuracy figures come from Pangram’s published claims and the University of Chicago working paper. We did not run our own scan of Substack posts for this article, so every number here belongs to the source linked next to it, and Reddit reports are labelled as reports.

Frequently asked questions

Vlad Ivanov

Runs Detection Drama, where he tests AI detectors and humanizers hands-on and tracks how platforms and schools use detection. LinkedIn

Sources

1. Engadget: Substack is adding an AI detection feature
2. 404 Media: Substackers say new AI detection tool is a witch hunt
3. SAN: Substack’s new AI-detection tool has some critics worried about false flags
4. Net Influencer: Substack launches AI detection tools with Pangram
5. Media Copilot: Substack AI detection with Pangram
6. Chicago Booth Review: Do AI detectors work well enough to trust?
7. Jabarian and Imas, BFI: Artificial Writing and Automated Detection
8. haverin on Substack: Substack AI detection: Pangram opt out

Last updated: October 9, 2026