Originality Detects AI-Written Text From ChatGPT and GPT-3

Since OpenAI opened up their ChatGPT service to the general public, it’s now nearly impossible to know if the text you’re reading online was written by a human or by an AI system pretending to be one.

Several tools help to detect text written by systems like ChatGPT, but all of them were trained on previous systems like GPT-2.

Originality AI is a new tool that can detect if text was written by ChatGPT, GPT-3, or the new GPT-3.5. Because the tool was specifically trained on output from these systems, it’s better than previously-available tools at detecting both AI-written text and run-of-the-mill plagiarism.

I tested Originality AI out with both human-written and machine-generated text. This article walks through my results.

Want to try Originality AI? Sign up with my link and use code UWriterReferral50 to get a special discount for readers of our blog.

How Originality AI Works

When AI-text generation systems write words, they’re essentially starting with a single word, and then predicting what the next word should be. They’re basically like very complex versions of your phone’s autocorrect.

That makes them good at fooling humans. But because they’re always predicting the next word, it also means they follow predictable patterns. Their text feels original to human readers. But a well-trained AI system can detect their telltale patterns, identifying AI written text.

Amusingly, it takes an AI system to catch an AI system! 

To work well, though, the AI detection system needs to be trained on lots of output from the AI writing system it’s trying to detect. Originality AI trained their system on output from GPT-2, GPT-J, GPT-NEO, GPT-3, and ChatGPT, so it’s specifically tuned to ferret out text from those systems.

Signing Up for the Service

Signing up for Originality AI is easy. Head to the Originality AI site, and sign up for an account. Again, you can use our special code UWriterReferral50 to get a discount.

Once you’ve signed up, you can purchase credits to use the service. They’re inexpensive–$10 buys you 1,000 to 2,000 credits depending on the discount code you use, which is enough to scan about 20,000 words. 

Originality AI clearly intends for their tool to be used at scale by major publishers or website owners who want to protect their site from AI-written content and its potentially disastrous SEO consequences.

My Experiments With Originality AI

I tested Originality AI extensively with both AI-generated and human-written text. Here are the results of my experiments.

Detecting Text from ChatGPT

ChatGPT is available to the general public, which means anyone can use it to write a blog post, or more concerningly, an academic paper. For that reason, I decided to start by testing how Originality would detect text written by ChatGPT.

I went to ChatGPT, and gave it the following prompt.

Write a paragraph about Bichon Frises and how often they need to be bathed.

It responded with:

Bichon Frises are a small breed of dog that are known for their fluffy, white coats. These dogs have a playful and cheerful personality, and they are known for being great companions. In terms of grooming, Bichon Frises typically need to be bathed every 4-6 weeks in order to keep their coats looking clean and fluffy. It is important to use a mild dog shampoo when bathing a Bichon Frise, as their skin can be sensitive. Regular brushing and grooming is also necessary to keep their coats free of tangles and mats. Overall, Bichon Frises are low-maintenance dogs that are easy to groom and care for.

It’s not going to win a Pulitzer, but that feels like something a human could have written.

I then went to the Originality AI interface and entered the text. The system took about 10 seconds to scan it.

Immediately, it responded with its verdict. Originality was 99% certain the text was written by AI. That’s exactly on point and a great first result! Originality seems to have no trouble detecting text written by ChatGPT.

Detecting ChatGPT Text That’s Trying to Hide

Next, I decided to test the system with text that’s trying to avoid detection. For that, I gave ChatGPT a version of my original prompt, but with a twist.

Write a paragraph about Bichon Frises and how often they need to be bathed, in a style that's very different from the writing of an AI system like GPT-3.

Here’s the result:

Ah, the beloved Bichon Frise! These little balls of fluff are the life of the party, always ready for a good time. With their fluffy white coats and playful personalities, it’s no wonder they’re such a popular breed. But as with any pup, regular grooming is a must to keep them looking and feeling their best. When it comes to bathing, the Bichon Frise is a bit more low-maintenance than some breeds. These little guys only need a good scrub-down every 4-6 weeks, so you don’t have to worry about setting aside hours of your day for bath time. Just be sure to use a mild dog shampoo to avoid irritating their sensitive skin. And don’t forget the brushing! Regular grooming will keep their coats free of tangles and mats, so they can continue to be the fluffy, happy dogs we all know and love.

It’s definitely voicier! In this case, Originality AI was a bit less certain, but still assigned an 81% certainty that the text is AI-generated! That’s pretty solid.

Detecting Human-Written Text

So Originality does a good job of detecting AI-written text. But how do we know it’s not providing a bunch of false positives?

As another test, I handed Originality some text that I wrote myself, from one of my recent Medium articles.

ChatGPT is blowing up. Twitter is inundated with screenshots of the app, coding sites like Stack Overflow are already banning answers produced with it, and over 1 million people have played with it. It’s a sensation. As a professional AI researcher, I wouldn’t have called that. ChatGPT is trained specifically to act as a chat bot, but fundamentally it’s using the same GPT-3 technology that’s been available for over two years now. What ChatGPT 3 demonstrates — moreso than impressive technology — is the crucial role that access plays in making breakthroughs truly usable. By packaging GPT-3 in a way that normal people can use, OpenAI has finally made people sit up and realize the incredible power of today’s AI. This is nothing new; we think of Thomas Edison as the inventor of the lightbulb, not because he actually invented it, but because he successfully brought it to market and turned it into something that normal people could understand.

The system quickly responded with 100% certainty that the text is original. It also got that the text was plagiarized, and included a link to the Medium article. It’s nice to have a plagiarism checker and AI detector bundled into one service.

I also tested the system by providing it with AI-written text from an example story that we published on this blog. It correctly determined that the text was both 100% AI-generated and 100% plagiarized.

Detecting AI Text from GPT-3.5

Clearly, Originality can detect AI-written text from ChatGPT. But what about text from the more powerful, commercial-grade GPT-3 or GPT3.5?

To see how it would do, I gave the same Bichon Frise prompt to GPT-3.5. It produced this output:

The Bichon Frise is a sweet and loyal dog breed that loves to be around its family. It’s a small breed with a big personality, and its long, fluffy coat of white fur makes it an irresistible companion. In terms of grooming, Bichon Frises need to be bathed regularly in order to keep their coats clean and healthy. The frequency of baths can vary depending on how often the pup is playing outside and how dirty their coat gets, but generally, they should be bathed every two to four weeks. In between baths, regular brushing is recommended to keep their coat looking its best and to prevent matting.

Originality wasn’t fooled! It gave a 97% chance that the text is AI-generated.

Detecting AI Text Rewritten by a Person

I ended my testing by giving Originality AI a harder task–detecting AI generated text that had been rewritten by a human.

To do that, I took the AI-generated paragraph about Bichons from GPT-3.5 and edited it myself, just as I might edit a paragraph submitted by a human writer. I made it flow better, eliminated some mistakes, and gave it a stronger voice. I also deliberately included a typo–something AI systems rarely include!

The Bichon Frise is a sweet and loyal dog breed that loves being around humans. Bichons are a small breed with a big personality! The Bichon’s long, fluffy coat of white fur makes it an irrestisibly-cute companion. What about grooming that coat, though? Bichon Frises need to be bathed regularly to keep their coats clean and healthy. How often depends on how often the pup is playing outside, and how dirty their coat gets. Generally, though, they should be bathed every two to four weeks. In between baths, you’ll want to brush your Bichon’s coat keep it looking its best, and to prevent the dreaded Bichon mats.

This time, Originality AI was fooled. It gave the text a 100% chance of being original, despite the fact that I kept about half of the AI-generated words in there.

Results

Overall, Originality AI words impressively well.

• It can easily detect text written by ChatGPT and even the commercial-grade GPT-3.5 with almost no ambiguity.
• Even when I had GPT-3.5 deliberately try to hide its tracks, Originality still gave an 80%+ certainty that its text was indeed AI-generated.
• Perhaps just as importantly, Originality AI identifies human-written text correctly. I found that giving it enough text was important; if I dropped the input to just a few sentences, it was less certain. But given enough human text, it avoided a false positive.
• The only area where Originality AI struggles is with AI-generated text that’s been rewritten by a person. Of course, this kind of text opens up some tough questions. Once a person edits AI-written text, is it really still AI-written? Or has it become human-generated at that point?

That’s a tough philosophical question without a clear answer. But what is clear that is Originality AI can easily detect the kind of mass-generated, unedited AI text that’s often used to create synthetic articles, papers, or websites.

That’s a powerful capability. The plagiarism detector worked great, too.

Other Features

Originality has some other nice features, too. The system includes an API, so publishers can build AI text-detection into their applications or content management systems at scale. 

The system also allows multiple team members to access an account, so editing teams or writing teams can share credits. 

I like that the tool keeps a history of your inputs and outputs, too, so you can always go back and see the text that you entered and the results the system returned.

Apparently, Originality will soon release a tool that can scan and entire website, looking for AI-generated text! Get ready to scan your favorite news sites to see how much they’ve been outsourcing to machines!

Uses for Originality AI

Based on my testing, I can see several uses for Originality AI.

Publishing

Publishers could use the platform to ensure that their writers aren’t submitting AI-generated content. Buying content is expensive, and it’s a badly-kept secret that many budget copywriters are actually using AI.

Since this can have devastating effects on SEO and reader trust, it’s important for publishers to find out of their writers are secretly sending AI-written content. Originality would work wonders for that.

I’m planning to use it to vet content for my own niche sites, to ensure that I’m not mistakenly buying valueless AI content.

Academia

Cheating in a class by submitting an AI-written essay or term paper is now incredibly easy, the Atlantic reports. Many professors already use plagiarism checkers on student papers. Now, Originality gives them the option to check for both plagiarism and AI-written content.

Of course, sensitivity is warranted, since the system isn’t perfect. Sometimes students write something so formulaic that it just feels like it was written by a robot!

But if a professor noticed a concerning result from Originality, they could at least have a conversation about it with the student.

Content Consumers

This is one of the coolest use cases. Right now, everyday people browsing the web have no idea if they’re reading something written by a person, or written by AI and published under the person’s byline.

Especially with its super low price point, Originality provides a tool that blog readers, news consumers and others could use to check if the content they’re reading is really human-generated.

Just read an article about a medical condition? Why not confirm whether it was written by a real person, or an AI robot pretending to be a doctor?

Suspect that a vitriolic political article was written by a bot? Plug it into Originality and find out. I like that the system can empower everyday readers to critically analyze and protect themselves from AI text.

How to Try Originality AI

Based on my testing, I’m impressed by Originality AI. I like it enough that I spoke with its creators, and they kindly extended a discount for my readers. 

To use it, head to their site using my link and enter our code UWriterReferral50. You’ll get a deep discount.

Give Originality a try, and see how it fits into your own content workflow.