All posts

AI detectors are basically just Old English detectors.

They can't tell if a robot wrote something, but they're great at catching anyone who writes like it's 1787!

5 min read

On this page
  1. The Founding Fathers
  2. How They Actually Work
  3. Who Actually Gets Flagged
  4. Even OpenAI Gave Up
  5. This Isn’t Just A Meme
  6. My Take!

Let’s start with a few simple facts:

I’m not a linguist, and I have never built an AI detector!

…but I’ve pasted enough of my own writing into these things to have opinions, and the more I read about how they actually work, the more I’m thinking that we’ve all been calling them the wrong thing.

They aren’t AI detectors. They aren’t even machine detectors. They’re formal English detectors. Old English detectors, if you want to be specific about it. (Not actual Old English. Beowulf would probably break them. I mean the buttoned-up n’ proper kind.)

The Founding Fathers

Back in 2023, screenshots started going around of people pasting the US Constitution into GPTZero. The verdict?

“Your text is likely to be written entirely by AI.”

ZeroGPT agreed with this conclusion. Passages from the Bible got flagged too.

So either the Founding Fathers had a ChatGPT subscription, or something else is the matter.

Ars Technica dug into why AI detectors think the US Constitution was written by AI, and the answer is kind of funny.

How They Actually Work

Most detectors tend to lean on two numbers.

  • Perplexity is how surprised a language model is by your text. Predictable word choices mean low perplexity. Weird ones mean high perplexity.
  • Burstiness is how much your sentences vary. People tend to mix long sentences with short ones. Models tend to keep things even.

Low perplexity? Low burstiness? Both?! The detector says that’s AI.

Here’s the problem. To a language model, the Constitution is about the least surprising text in the English language. Edward Tian, who created GPTZero, put it like this:

“The US Constitution is a text fed repeatedly into the training data of many large language models.”

The model has basically memorized it. Of course it isn’t surprised.

So the detector ends up asking “does this sound like something I’ve read a million times?” And formal, polished, careful writing sounds exactly like that. Anything from legal documents to academic papers. Heck, even old textbooks. The kind of letters your grandparents wrote.

Who Actually Gets Flagged

This is where it stops being funny.

In 2023, researchers at Stanford ran essays through seven popular detectors (paper, HN discussion). First they tried essays written by US eighth graders, and the detectors were close to perfect. Then they tried 91 TOEFL essays, written by people taking an English test in their second language.

Essays Flagged as AI
US eighth graders Almost never.
Non-native English speakers (TOEFL) 61% of the time, on average.
The same TOEFL essays by at least one of the seven detectors 97%

97%… yeeeaaah.

When you’re writing in your second language, you stick to the words and sentence shapes you’re sure about. Safe, common, predictable. Which is exactly what a detector reads as “robot.”

Then there’s “delve.” In 2024, Paul Graham said a cold email using the word “delve” was a sign it was written by ChatGPT. A good few Nigerian writers pointed out that “delve” is a totally normal word in Nigerian English, where a more formal register is just how people write. One theory, which Simon Willison rounded up, is that AI companies hire a lot of people in Nigeria to rate model answers, so the models might have picked up that style from them in the first place.

So “sounds like AI” can really just mean “learned English somewhere else.”

And then there’s em dashes. At some point last year, the internet decided em dashes mean you used ChatGPT. Writers who’d used them for years started rethinking their own punctuation! A punctuation mark that’s been around for centuries… retired because a chatbot likes it.

(You may notice there isn’t a single em dash in this post. I’m not taking any chances!)

Even OpenAI Gave Up

In January 2023, OpenAI released its own AI text classifier. On July 20, it was gone, “due to its low rate of accuracy” (HN discussion). By their own numbers, it caught AI-written text 26% of the time, and it flagged real human writing as AI 9% of the time.

The company that makes ChatGPT couldn’t reliably tell when ChatGPT wrote something. A couple of months later, their FAQ for educators answered the question “Do AI detectors work?” with, more or less, “not in our experience.”

This Isn’t Just A Meme

Last October, the ABC reported on Australian Catholic University. The university recorded nearly 6,000 suspected academic misconduct cases in 2024, around 90% of them about AI, and a lot of them leaned on Turnitin’s AI detector. Turnitin itself has warned that its score shouldn’t be the only basis for action against a student.

One of those students was Madeleine, a nursing student whose essay came back as 84% AI. It took six months to clear her. For all six of those months, her transcript said “results withheld,” and she says that’s part of why she didn’t get a graduate job.

To be fair, ACU says those numbers were overstated, and around a quarter of the referrals were dismissed once they were actually investigated. They also stopped using the AI detector in March 2025. But six months is a long time to wait for someone to admit the computer was just guessing.

My Take!

I get why teachers want something like this, but it just simply doesn’t exist yet. I get it, nobody wants to grade a stack of essays that a random chatbot wrote. But a tool that mostly catches people who write carefully, or who learned English as a second language, isn’t solving that problem. If anything, it makes that problem worse. For what it’s worth, here’s where I’ve landed on the matter:

  • Ask about the process. Drafts, notes, version history, or just a five-minute conversation can tell you way more than a percentage.
  • Keep your drafts! If you write anything that might get run through one of these, version history is your best friend. Google Docs keeps it automatically. (Or maybe put your essays in Git, I won’t judge.)
  • Don’t dumb your writing down. Please don’t start avoiding good words or semicolons because a tool might not like them. Trust me when I say that’s the detector’s problem!

Good writing has always been a little bit predictable. That’s kind of the whole point of grammar. Anywho, if this post gets flagged as AI, I’m (respectfully) choosing to take it as a compliment! :’)

Type a command and press Enter. Type help for a list. Tab completes.