A new report from plagiarism detector Copyleaks found that 60% of OpenAI’s GPT-3.5 outputs contained some form of plagiarism.
Why it matters: Content creators from authors and songwriters to The New York Times are arguing in court that generative AI trained on copyrighted material ends up spitting out exact copies.
The individual GPT-3.5 output with the highest similarity score was in computer science (100%), followed by physics (92%), and psychology (88%).
And that’s why this claim is mostly bullshit. These use cases are all sciences, where the correct solution is usually the same or highly similar no matter who writes it. Small snippets of computer code cannot be copyrighted anyway.
Not surprisingly, softer subjects like “English” and “Theatre” rank extremely low on this scale.
Not to mention that a response “containing” plagiarism is a pretty poorly defined criterion. The system being used here is proprietary so we don’t even know how it works.
I went and looked at how low theater and such were and it’s dramatic:
The lowest similarity scores appeared in theater (0.9%), humanities (2.8%) and English language (5.4%).
Yeah, anyone who has written a thesis knows those tools are bullshit. My handwritten 140 page master’s thesis had a similarity index of 11%.
Pun intended?
So, if the Ai gives you a correct answer to a science question, it’s “infringing copyright” and if it spits out a bullshit answer, it’s giving you wrong, and unsupported claims.
Right? Nod doubt that output can be similar to training data, and I would believe that some of it is plagiarism, but plagiarism detectors are infamous among uni students for being completely unreliable and flagging pronouns, dates and citations. Until someone can go “here’s an example of actual plagiarism” (which is obvious when pointed out), these claims make no sense.
If it’s plagiarizing, so are Google search results summaries.
It’s not like it doesn’t cite where it found the data.
Eh, kinda. It’s not like a science paper is just going to be an equation and nothing else. An author’s synthesis of the results is always going to have unique language. And that is even more true for a social science paper.
Are those “best matches” paper-sized, or snippet-sized?
Article mentioned 400-word chunks, so much less than paper-sized.
I think the issue is more about HOW they wrote it, rather than WHO wrote it.
You can’t write a paper covering scientific topics without plagiarism. A human would be required to. Generative AI should be held to at least as high of a standard.
Turns out ChatGPT isn’t writing a scientific paper though, it’s conversing with the user.
If it’s regurgitating other people’s work then it needs citations.
This looks like an ad. They go on about what their proprietary detection method found without any details about how it came to these conclusions or even how they generated the test data. They give 0 actual examples for any of their claims.
Here’s the original blog post the article is referencing: https://copyleaks.com/blog/copyleaks-ai-plagiarism-analysis-report
Yep, Axios straight-up printed an ad as news.
They should show a small, but representative sample of questions they gave it.
Also they should compare the scores to similarity scores for a flesh and blood smart human that answers the questions.
Well, I tried it. So here’s an example.
this may soon be a thing of the past as
This fragment was flagged as plagiarism.
Don’t worry, It’s only piracy if a poor person does it.
“Plagiarism detection company claims LLM conditions plagiarism according to their detector.”
I wonder how many student written essays also contain ‘plagiarism’ according to their tool.
Probably very few. The bias for these companies is in false negatives, not false positives, since false positives create controversy when students appeal a ruling.
The bias here was certainly to come up with a lot of false positives for advertising; kinda like anti-virus companies do it.
100% iirc, there are only so many ways to write about how the blue curtains indicate the character is feeling depressed or something.
Ai outs ai. Also, haven’t these ai anti-plagiarism tools shown to have very high false positive rates?
Isn’t that basically what the current LLM AI fundamentally does? Just digests a bunch of text and gives a summary back in response to queries?
No. It’s not really clear what LLMs do, but it certainly depends on context.
What they fundamentally do is continue a text. That’s what they were originally trained to do. Then they were fine-tuned to continue a chat log or respond to an instruction. To be able to do that, they have learned a lot. Unfortunately, we do not know what.
If you ask for a summary of some text, it will give you one; regardless of whether the text even exists.
The summary could be one written by a human that it has memorized. Or it could be complete nonsense, that it is making up on the fly. You never know.
One AI company throwing accusations at another AI company, and the evidence on both sides is to point their fingers at their own black-box LLMs like they’re magic…
10% are bullshit legal case citing made up for lawyers who are lazy
Only 60%?
59.7%
ok, so? plagiarism is a meaningless, tenuous call that can be avoided simply with some quotation marks and a link. isn’t this supposed to unite humanity’s knowledge rather than nitter and ditter around about meaningless technicalities? it’s not writing a fucking paper (and if you copy and paste it for a paper that’s already plagiarism anyway)
if it had to remember a citation for everything it knew, it’d only be able to remember half as much information because its memory would be cluttered with useless citations that you could easily find by googling if you really cared to know. most people just want quick facts
Removed by mod
