How can you tell if people can accurately identify AI generated text?
If a person reads AI generated text and does not notice, they by definition will not know about it.
There have been numerous cases of people accessing human created content as being AI.
There are instances where it seems relatively uncontroversial that it is AI generated, but without knowing both the amount of AI content people are exposed toand the amount that they register I don't think you can draw a conclusion of the overall state.
I don't think this is about edge cases where someone has successfully disguised the writing to some degree: the current crop of LLMs have some pretty blatant (and frankly annoying) habits by default, ones that are hard to miss once you have read a decent amount of their output. If I had to describe them broadly, I would say they are a collection of habits which are common in certain kinds of persuasive and emotive writing, but are usually applied way out of proportion to the topic at hand, which tends to make the result quite grandiose, overly dramatic, and tiring to read: a LLM will often write a TODO app README like it's a cross between a thriller novel, a political speech, and a bombshell news article. There's lots of specific tics (and just by sheer volume and uniformity almost any habit an LLM picks up is going to rapidly shoot into cliche regardless of its own merit) but this is the general effect which I think is objectionable independent of the source of the text.
I do think the sensitivity to it can vary a lot: it depends a lot on how much and how closely you read the text, and how much exposure you have to LLM writing. Certainly it seems like a lot of people just don't really notice, or at least don't care much.
That’s the aspect I’ve had a hard time articulating. Definitely the right comparison. It feels like some huge revelation has been had and it’s so consequential and it’s unbelievable that it’s happening right before your very eyes and wow aren’t you so insightful and pushing the boundaries of knowledge!?
That plus the “it’s not X, but why” nonsense makes it feel like some condescending parent is trying to lecture me but at least 30% of what they’re saying is probably made up.
> the current crop of LLMs have some pretty blatant
This is just "em dash redux." Except now we've moved on to accusing anyone who does "It's not X. It's Y." of being AI. In six months, it'll be "use of the word 'petrichor'" or something.
(a) believe this is human prose
(b) enjoy reading this prose
(c) would enjoy reading 100 READMEs like this.
As for invoking petitio principii and questioning other commenters' logical coherence [0], can you politely shove the argumentum ad Latinum up your ass?
I do think there is a tendency to over-index on one or two particularly straightforward tells, and for any given feature of LLM writing you can find places where people do also use that feature (they had to learn it from somewhere, and in a lot of cases it is good writing practice — for the context in which it is used). But I'm not talking about just that, but also the general tone issue: it's bad writing regardless because it's in most cases just not appropriate for the context it's been written in.
(TBH I think the biggest likelihood for false positives comes from heavy LLM users picking up their tics: it's a natural tendency and I've already seen a few cases where it seems like that has happened).
Idk, it's more like "your writing is cliché and I don't feel like reading it because I've already read something that sounded similar countless times and it wasn't worth the read". The source of the clichés being an LLM. And maybe now humans are writing the same way as LLM output, I still am not going to read all that, sorry. If I see a sea of clichés, I'm going the other way.
I'm also not reading pumpkin spice murder mysteries for a similar reason. I'm also not reading stories where everybody clapped. Actually, I'm already familiar with petrichor, so unless someone has surrounded the word "petrichor" with non-cliché prose, I'm also not going to read all that.
Some people are still losing their shit over em-dashes, with no other tells, and humans can't use the not x; y construction anymore either, regardless of any other merit to the writing.
LLM writing is verbose and meandering, but people are making a much bigger deal over this stuff than necessary for virtue signalling purposes. You don't want to read someone else's LLM writing? Get a summary of the page from yours. No time wasted, no pretentious posturing, and you don't make the error of assuming because the piece was written by an LLM that there was no thought put into the subject or there's no value in what is being communicated.
> You don't want to read someone else's LLM writing? Get a summary of the page from yours.
I am disinclined to take someone's error-prone machine generated text and run it through an error-prone summarizer. That's a waste of time (not to mention electricity), when all that needed to happen was for the original person to not be so fucking lazy and just write out his thoughts.
This is very handwavy and dismissive. It is pretty safe to assume that must of us catch it most of the time because the simple fact is so many people just copy and paste whatever the LLM outputs without even trying to edit it or mask that they used one. We’ve all seen so many examples of the exact same cadence and verbiage that we’ve learned how to identify it pretty reliably. The ones who are “slipping past us” are actually putting in the work make not just pasting raw LLM outputs, which is the real issue here. If somebody has edited it meaningfully after the fact then it’s not the same crime.
> It is pretty safe to assume that must of us catch it most of the time because the simple fact is so many people just copy and paste whatever the LLM outputs without even trying to edit it or mask that they used one.
This statement does not logically cohere. "We can spot it because so many people make it easy to spot." You don't see how this is just petitio principii in action?
There is no group of people who put enormous amounts of effort into not putting effort in to writing.
To fully disguise LLM prose, you'd have to rewrite it entirely, and if you were going to do that, you wouldn't be the sort of person to use it in the first place.
What is your point? It’s not that complicated. Obvious slop is obvious. Maybe there are some humans out there who sound like Claude but I’m not going to force myself through 900 slop blog posts on the off chance that one of them might actually be written by a human.
Maybe some people stop reading LLM slop purely because it violates their moral principles or whatever but most people bail out because slop is mentally painful to read. If you are a human and you write like today’s AI find a different writing style, not because reads like AI, but because it reads like shit.
If something is obvious then by definition it’s obvious. Additionally the formula is simple: AI + effort = writing we can generally tolerate (unless you’re just a bad writer). AI + no effort = writing most people can’t tolerate regardless of your skill as a writer.
LLM writing, unless someone puts in the time to improve it, typically follows the exact same patterns and favors the same words. You can’t reas a thread here without people talking about “Claude speak.” It is readily apparent, I do not need to show you a 10 year study with n=100,000 to make this point. The article isn’t hallucinating a problem, we are all nodding along because we all see it every goddamn day lmao. He even cited a study and makes a pretty strong case for why it’s a useful metric here in TFA.
If the AI writing is indistinguishable from human writing, then it is not lazy copy and pasting of AI outputs and isn’t just raw LLM output with no work done on it. So in that case it’s no longer a problem.
LLM’s cannot write in a natural, human way that distinguishes it from the typical LLM output on the first try. If they could, we wouldn’t have this problem. Maybe one day they will. Hell maybe it’ll even be next week. But currently they do not so I do not understand why we are having this discussion.
AI writing just means "writing I don't like" now. Just like Nazi means whatever and whoever I politically disagree with. Words have lost their meaning.
Just like "violence" can now mean speech you don't agree with, "genocide" means military action you don't agree with, and nobody seems to know what "woman" means anymore, Maybe the solution is to stop using words.
You're right that Nazi doesn't mean Nazi anymore. It means neonazi / white supremacist / white nationalist, which is a much broader group of people that, for some baffling reason, are under the impression that people don't care about their fascism and racism anymore.
Clearly not, or this wouldn't be something people discuss at all.
There are lots of people who belong to the above groups, sure, but at least here in Germany Nazi is now applied to basically anyone who doesn't vote green, it's ridiculous.
If a person reads AI generated text and does not notice, they by definition will not know about it.
There have been numerous cases of people accessing human created content as being AI.
There are instances where it seems relatively uncontroversial that it is AI generated, but without knowing both the amount of AI content people are exposed toand the amount that they register I don't think you can draw a conclusion of the overall state.