[Q/D] AI-generated WA proposals as plagiarism

[Simone Republic]

What's the cutoff criteria you would set for the detector's certainty? Because I got results of 17, 13, 5, and 0% on parts of my writing in May:
viewtopic.php?p=40613877#p40613877
Others have gotten 50%:
viewtopic.php?p=40610273#p40610273
We are not dealing with a system that can reliably avoid false positives, even if we haven't seen proof of a false positive "in the wild", so to speak. And as I said before I would not be comfortable with for instance an error rate of 5%.

OK, here's some statistics I generated using GPTzero.me (which conveniently is free and has a char count limit of 5,000, so fits the GA/SC count), exclusively on stuff that I wrote (and I am happy to affirm on a Bible that I have never used ChatGPT, otherwise it wouldn't be so plagued by grammatical errors):

SC:
On the Strangers Bar etc (the upcoming) vote: 15%
SC#482 (Mariah Carey parody): 0%

GA (recent new topics, not repeals):
GA#703 (consular protection): 0%
GA#702 (infant nutrition): 0%
GA#695 (dual citizenship): 1%
GA#690 (repeal end blood sports, IA wrote this): 1% (that's 93 words so the shortest in the sample)
GA#686 (identity documents): 0%
GA#681 (non-compete clauses): 1%

And to test out every other member of GenSec, since I have co-authored with all of them except D&B, and most of them are repeals:
Repeal Tin's challenging records (Wally): 17% (the preamble, which followed an established UN format, was identified as AI, the rest was not)
Repeal Legal Equality Act (Kenmoria): 0%
Repeal #401 (Magecastle and myself, this passed as #641): 14% (again, GPTzero.me put up a "text too short alert")

GA#686 includes texts originally from IA, GA#681 includes edits from Cretox, and original text from both IA and Sep. And that road-tested every member of Gensec (except D&B, unless he has an author credit that I don't know about).

Since GPTzero.me is a free service, anyone can test that my numbers above are as generated.

Based on the above, I have some pushback about "dealing with false positives".

I haven't tried GPTzero.me on any other author's resolutions since I think that's for them.

[Gruenberg]

It also returns 11% for the resolution Wrapper claimed clocked in at 50%.

[Picairn]

It also returns 11% for the resolution Wrapper claimed clocked in at 50%.

ZeroGPT, the checker that Wrapper was responding to, returns a 43.83% AI rate. So what do we do if one proposal returns such wildly differing results?

[Gruenberg]

Not use ZeroGPT?

[Picairn]

Not use ZeroGPT?

So how do you know which checker to use and which not to?

[Gruenberg]

Experience, I guess. Though to be honest, I didn't use any checker on the homework one, it was just obvious.

[Picairn]

Experience, I guess. Though to be honest, I didn't use any checker on the homework one, it was just obvious.

So just vibes, then. Not a serious solution.

[Second Dimetrodon Empire]

Well, we only can know for sure in this case because the author admitted it. That doesn't show anything about any other proposals

If the author doesn't admit it, the only evidence you'll have is faulty AI checkers. I'm leaning towards Dimetrodon's position than a blanket ban.

Considering how frequently accusations of using Chatgpt end up resulting in admissions, I do not see why proposals cannot be removed for plagiarism at least when we know for sure that Chatgpt was used. I can't imagine people falsely confessing to having used Chatgpt.

I am not talking about false confessions.

I am talking about those who use it yet don't confess at all. You may have encountered obvious cases but wait until you encounter a bunch of cases that aren't so obvious. Like those who may use chatGPT to write most of the proposal and then intertwine some of their own words in it to obfuscate it. Or it seems like it may be generated by AI but there is really insufficient information to make a grounded conclusion.

It also returns 11% for the resolution Wrapper claimed clocked in at 50%.

ZeroGPT, the checker that Wrapper was responding to, returns a 43% AI rate. So what do we do if one proposal returns such wildly differing results?

This ^^

Experience, I guess. Though to be honest, I didn't use any checker on the homework one, it was just obvious.

Yes, but in the future, there will be cases where it won't be.

[Gruenberg]

Yes, but in the future, there will be cases where it won't be.

OK, let's deal with those in the future, and for now shitcan the ones that are patently AI written.

[Simone Republic]

It also returns 11% for the resolution Wrapper claimed clocked in at 50%.

ZeroGPT, the checker that Wrapper was responding to, returns a 43.83% AI rate. So what do we do if one proposal returns such wildly differing results?

ZeroGPT also returned 0% on my last two GA resolutions, GA#702 and GA#703. I need to get an account etc., will post the results of my own stuff later. (Since that's public, anyone can replicate the results, I haven't posted them directly from the web site).

[Picairn]

Someone with enough time and statistical knowledge should gather a team and run 100 random GA resolutions through 5 or so AI checkers to see which one is less likely to return a high AI rate. I still think this is sketchy but statistical inferences are still more legit than going with vibes.

Edit: To be explicit, I mean *historical* GA resolutions.

[Second Dimetrodon Empire]

Yes, but in the future, there will be cases where it won't be.

OK, let's deal with those in the future, and for now shitcan the ones that are patently AI written.

Well, as Picairn said already, vibes and obvious cases aren't even much of a solution now. And potential cases, are you going to demand that mods copy and paste multiple proposals into multiple AI checkers?

Again, enforcement isn't as easy as you think it is. The main reason why you know you were right was simply because it was obvious that time and the "author" admitted to it. The "future" I mentioned is probably much closer than you think it is as well.

The best option is to vote suspicious proposals down, like what I said earlier. You and most of the community think it is AI generated, then vote it down. Making it a rule will be at best, difficult to enforce, and at worst, bad results can result in innocent authors receiving warnings from moderation. It will also create a lot more work for the moderators.

Compare that to the stakes of a mass vote down where the proposal doesn't pass and the author is simply mildly annoyed in the case of a mistake.

[Simone Republic]

Someone with enough time and statistical knowledge should gather a team and run 100 random GA resolutions through 5 or so AI checkers to see which one is less likely to return a high AI rate. I still think this is sketchy but statistical inferences are still more legit than going with vibes.

Edit: To be explicit, I mean *historical* GA resolutions.

Probably should ask IA (I will ping him to ask on GA Discord), since the software to archive GA resolutions is on his Github. And he definitely has enough statistical knowledge, even though I (somewhat) choke on his assertion that Python is better than R.

[Gruenberg]

OK, let's deal with those in the future, and for now shitcan the ones that are patently AI written.

Well, as Picairn said already, vibes and obvious cases aren't even much of a solution now. And potential cases, are you going to demand that mods copy and paste multiple proposals into multiple AI checkers?

No, although I would assume checking for plagiarism already involves copy+pasting into a search engine or similar, so I don't really see the difference.

As for voting proposals down, that's never been used as a substitute for rules enforcement.

[Picairn]

No, although I would assume checking for plagiarism already involves copy+pasting into a search engine or similar, so I don't really see the difference.

The difference is that AI doesn't regurgitate whole resolutions verbatim, it writes in its own distinct way that may or may not be detected by AI checkers. It is very easy to detect plagiarised proposals, but not AI-written ones. It gets even harder if the author writes a resolution then gets AI to improve the text, the result would be a mix of human and AI-written speech that would be difficult to pin down correctly.

[Vistulange]

Someone with enough time and statistical knowledge should gather a team and run 100 random GA resolutions through 5 or so AI checkers to see which one is less likely to return a high AI rate. I still think this is sketchy but statistical inferences are still more legit than going with vibes.

Edit: To be explicit, I mean *historical* GA resolutions.

Probably should ask IA (I will ping him to ask on GA Discord), since the software to archive GA resolutions is on his Github. And he definitely has enough statistical knowledge, even though I (somewhat) choke on his assertion that Python is better than R.

Python is better for tasks which aren't data cleaning or data analysis. I say this as an ardent defender of R. What gets me annoyed is Python users thinking Python is better at everything. I wouldn't use R for this task, I'd use Python!

But anyway, it might be a wild goose chase that you're attempting. A reliable LLM-detector would entail building something as sophisticated as those LLMs themselves due to the way in which these LLMs create outputs. Vibes, honestly, might even be a better way to go about it, practically speaking.

[Attempted Socialism]

What's the cutoff criteria you would set for the detector's certainty? Because I got results of 17, 13, 5, and 0% on parts of my writing in May:
viewtopic.php?p=40613877#p40613877
Others have gotten 50%:
viewtopic.php?p=40610273#p40610273
We are not dealing with a system that can reliably avoid false positives, even if we haven't seen proof of a false positive "in the wild", so to speak. And as I said before I would not be comfortable with for instance an error rate of 5%.

OK, here's some statistics I generated using GPTzero.me (which conveniently is free and has a char count limit of 5,000, so fits the GA/SC count), exclusively on stuff that I wrote (and I am happy to affirm on a Bible that I have never used ChatGPT, otherwise it wouldn't be so plagued by grammatical errors):

SC:
On the Strangers Bar etc (the upcoming) vote: 15%
SC#482 (Mariah Carey parody): 0%

GA (recent new topics, not repeals):
GA#703 (consular protection): 0%
GA#702 (infant nutrition): 0%
GA#695 (dual citizenship): 1%
GA#690 (repeal end blood sports, IA wrote this): 1% (that's 93 words so the shortest in the sample)
GA#686 (identity documents): 0%
GA#681 (non-compete clauses): 1%

And to test out every other member of GenSec, since I have co-authored with all of them except D&B, and most of them are repeals:
Repeal Tin's challenging records (Wally): 17% (the preamble, which followed an established UN format, was identified as AI, the rest was not)
Repeal Legal Equality Act (Kenmoria): 0%
Repeal #401 (Magecastle and myself, this passed as #641): 14% (again, GPTzero.me put up a "text too short alert)

GA#686 includes texts originally from IA, GA#681 includes edits from Cretox, and original text from both IA and Sep. And that road-tested every member of Gensec (except D&B, unless he has an author credit that I don't know about).

Since GPTzero.me is a free service, anyone can test that my numbers above are as generated.

Based on the above, I have some pushback about "dealing with false positives".

I haven't tried GPTzero.me on any other author's resolutions since I think that's for them.

My problem with your pushback is actually explained in the percentages I highlighted above. 14, 15, and 17%? Those are too high to rely on for my taste. If we consistently got 0% and 1% for non-AI resolutions, and 99% and 100% for admitted AI-written resolutions, that would be acceptable for me to automate, but we don't. In addition, as Gruenberg said that Wrapper's resolution returned 11% for them (As opposed to the 43% for Picairn and 50% for Wrapper originally) we also can't rely on just a number, we would have to decide which detector we use and what version of that detector since that too changes over time.

With all the admitted or alleged instances of plagiarism the text has been off enough that it would never pass even if it wasn't plagiarism, so I maintain that the best solution is one akin to the age rule: If someone admits to plagiarism (Or posts something that is sufficient proof) then that is a cause for action. Resolutions that are suspected of being AI-generated won't pass anyway and often die with hardly any endorsements.

[Pacific Haven]

The homework one was easily identifiable as AI. The author bumped it after 13 minutes, had written no proposals before, and submitted it without looking on the forum. It was in the queue by the time it reached the forum.

Other ones will obviously be harder to identify (but, generally, people who use AI aren't very smart and don't usually cover the tracks, like the homework one), but remember the homework one was checked by an AI checker and I found that it was 80% written by AI.

Better than the 50% some of you are suggesting.

[Pacific Haven]

OK, here's some statistics I generated using GPTzero.me (which conveniently is free and has a char count limit of 5,000, so fits the GA/SC count), exclusively on stuff that I wrote (and I am happy to affirm on a Bible that I have never used ChatGPT, otherwise it wouldn't be so plagued by grammatical errors):

SC:
On the Strangers Bar etc (the upcoming) vote: 15%
SC#482 (Mariah Carey parody): 0%

GA (recent new topics, not repeals):
GA#703 (consular protection): 0%
GA#702 (infant nutrition): 0%
GA#695 (dual citizenship): 1%
GA#690 (repeal end blood sports, IA wrote this): 1% (that's 93 words so the shortest in the sample)
GA#686 (identity documents): 0%
GA#681 (non-compete clauses): 1%

And to test out every other member of GenSec, since I have co-authored with all of them except D&B, and most of them are repeals:
Repeal Tin's challenging records (Wally): 17% (the preamble, which followed an established UN format, was identified as AI, the rest was not)
Repeal Legal Equality Act (Kenmoria): 0%
Repeal #401 (Magecastle and myself, this passed as #641): 14% (again, GPTzero.me put up a "text too short alert)

GA#686 includes texts originally from IA, GA#681 includes edits from Cretox, and original text from both IA and Sep. And that road-tested every member of Gensec (except D&B, unless he has an author credit that I don't know about).

Since GPTzero.me is a free service, anyone can test that my numbers above are as generated.

Based on the above, I have some pushback about "dealing with false positives".

I haven't tried GPTzero.me on any other author's resolutions since I think that's for them.

My problem with your pushback is actually explained in the percentages I highlighted above. 14, 15, and 17%? Those are too high to rely on for my taste. If we consistently got 0% and 1% for non-AI resolutions, and 99% and 100% for admitted AI-written resolutions, that would be acceptable for me to automate, but we don't. In addition, as Gruenberg said that Wrapper's resolution returned 11% for them (As opposed to the 43% for Picairn and 50% for Wrapper originally) we also can't rely on just a number, we would have to decide which detector we use and what version of that detector since that too changes over time.

With all the admitted or alleged instances of plagiarism the text has been off enough that it would never pass even if it wasn't plagiarism, so I maintain that the best solution is one akin to the age rule: If someone admits to plagiarism (Or posts something that is sufficient proof) then that is a cause for action. Resolutions that are suspected of being AI-generated won't pass anyway and often die with hardly any endorsements.

Again, I'll repeat my above point- 17% is nothing. You're scared of a false positive at 17% (or even 43-50%), while we got an actual positive at 80%?

[Reventus Koth]

All this back and forth about AI detectors is exhausting and pointless. You might as well be talking about how well each of your preferred brands of snake oil work.

Instead of trying to fabricate some complicated solution to an ever-increasingly complex problem, how about you just campaign against shitty proposals? If a proposal is convincing enough to clear the high standards of the WA community and its text isn't obviously copied from anywhere you can tell, then genuinely why give a shit if it's AI-assisted or not? You all seem to have a pretty good grasp on when a grifter is trying to slam through a 0 effort badge hunt, trust yourselves over snake oil.

[Pacific Haven]

All this back and forth about AI detectors is exhausting and pointless. You might as well be talking about how well each of your preferred brands of snake oil work.

Instead of trying to fabricate some complicated solution to an ever-increasingly complex problem, how about you just campaign against shitty proposals?

You clearly have no understanding of how the GA works. The "campaign against shitty proposals" is already a thing. It's called "being declared illegal for obviously breaking one of the proposal rules." That's not part of the discussion or really being contested.

If a proposal is convincing enough to clear the high standards of the WA community and its text isn't obviously copied from anywhere you can tell, then genuinely why give a shit if it's AI-assisted or not?

"Oh, look at me, I drafted a great GA resolution! It was written entirely by AI. I have a grasp of legal concepts and how to legislate in a realistic way! They don't. Look at my shiny badge and prestige in the hallowed halls of the GA!"

Because, for real/non-AI authors, how frustrating would it be if someone used AI to become better than you at writing something which they didn't actually write?

If a proposal is convincing enough to clear the high standards of the WA community and its text isn't obviously copied from anywhere you can tell, then genuinely why give a shit if it's AI-assisted or not?

You're either using loaded language to justify a point that isn't being contested, or fail to understand what AI is. ChatGPT, for example, is a glorified search engine- it is copying from somewhere else, and by extension so are you. And there's a difference between "AI-assisted" and "AI-written".

[United Calanworie]

How do we draw a line between AI “assisted” and AI “written?”

[Pacific Haven]

How do we draw a line between AI “assisted” and AI “written?”

"AI-written" is when you put in a prompt, e.g. "Write a resolution on X", and it spews out the content, and you hand it in.

"AI-assisted" is when the prompt is "How does this concept work?", and use that to write your resolution. For example, on a resolution on spacecraft, you might ask "How does a spacecraft work?", and ChatGPT tells you.

AI-written: You have little to no knowledge on a topic, but still want to write a resolution.
AI-assisted: You have a clear outline of what you want, but you can't write it in clear enough detail.

Of course there's the issue of how you know. Generally, if you put in your first draft without AI, and then use it- along with the community- to improve your technical details, there's no way someone would suspect you of using AI fully. If someone hands in a draft on spacecraft with full technical details, down to the physics (I have no knowledge of spacecraft whatsoever), we should be able to question them further on what they know.

[Orcuo]

I personally could care less if AI writes a proposal.

[Pacific Haven]

I personally could care less if AI writes a proposal.

GA authors care

(If you have nothing to add, just don't reply? Like this is still moderation)