Government change does not stick

[Rocain Founder]

I think there is a problem with my puppet Cain M Ca0 v0. That nation's initial page claims the government type is "Democratic Socialists", but the National Happenings on the same page twice notes that 'Cain M Ca0 v0 was reclassified from "Democratic Socialists" to "Authoritarian Democracy"', these announcements being separated by 5 hours (and related to different issues).

I can't think of any way this could be reasonable.

Edit: I should probably mention, in case it is relevant, that these issues were answered via the API.

[Trotterdam]

Based on that nation's current Civil Rights value (35.11), Democratic Socialists should be the correct category. Perhaps Civil Rights wasn't adjusted properly in response to the issues?

[Rocain Founder]

Since I posted this note, I have done some more research on the situation, and will spell out what happened more precisely. I believe that there is evidence of at least two events which should be impossible. Whether they have a single cause I do not know. I believe the chances are quite good that these phenomena occurred due to use of the API.

On July 14, around 2:30 EST, my script apparently was told, via the API, that the government type of Cain M Cao v0 was Democratic Socialists. It then answered issue #645, choosing option 3. According to the logs kept by my script, its civil rights stat dropped from 41.44 to 34.36, and the message ‘Cain M Cao v0 was reclassified from "Democratic Socialists" to "Authoritarian Democracy".’ was returned. An API inquiry confirmed that the government type was now Authoritarian Democracy. Other than the change-of-government message, I have no confirmation independent of my script of any of this.

Later that day, around 6:30 EST, I ran my script again. The API told the script that the government type was Authoritarian Democracy, but when I used the API to answer issue #458 with option 2, its civil rights stat dropped from 34.36 to 34.15, and I received the message ‘Cain M Cao v0 was reclassified from "Democratic Socialists" to "Authoritarian Democracy".’ once again. This is obviously wrong. Either the API was incorrect when it indicated the government type was already Authoritarian Democracy, or the change-of-government message was incorrect. The API call after answering the issue to retrieve the government type once more indicated it was Authoritarian Democracy.

At this time, I happened to notice the weirdness, logged in to the nation, and observed that the National Happenings reflected both government type change messages, which in themselves are signs of a problem, unless the government type could spontaneously change without an issue being answered. (I think this ought to be impossible since the target nation is not in the WA, and the only issue-related stats which should change without the intervention of an unanswered issue are those related to a nation’s population, and I have no reason to think that “civil rights” is such a stat.)

At the same time that I observed the National Happenings, I also observed that the nation’s start page displayed the government type of this nation as Democratic Socialists, despite a National Happening that said it had become an Authoritarian Democracy. This was on the same day as the day of generation of the two change-of-government messages. So the game proper and the API differed on what the government type was.

I have not answered an issue again for this nation. But yesterday (July 18), I used the API to inquire what its government type was. (I should have thought of doing this earlier. Sorry.) This time, the API told me it was Democratic Socialists, which a quick login also confirmed. So somewhere between the evening of the 14th and the 18th, the API changed its mind about what the government type was. Again, this should not be possible as far as I can see. I also note that the value the API gave me for the civil rights stats (34.15) differs from the value Trotterdam observed (35.11). I have made no independent attempt to verify Trotterdam’s observation, but if correct it is another indicator that something is messed-up with either this nation or some of the code that processes it.

Pretty clearly, there are one or more bugs somewhere, probably in the API, and probably in particular with the civil rights stat, support for which was recently changed. Of course, I can’t even begin to guess at the nature of the problem(s) beyond this mysterious evidence.

I am refraining from answering any more issues for this nation to avoid possibly muddying the water and making investigation any more difficult.

[Trotterdam]

Both of the following sources report the 35.11 value:
https://www.nationstates.net/nation=cain_m_ca0_v0/detail=trend/censusid=0
https://www.nationstates.net/cgi-bin/api.cgi?nation=cain_m_ca0_v0&q=census&scale=0&mode=score

[Rocain Founder]

I just wanted to post a minor update on this problem. The script which made the API calls that showed these problems with government type transitions manages issue answering for about 70 of my puppets, many of which answer more than once per day. About a week ago, I added code to the script to detect and complain about any problems with government type changes similar to the ones I reported above. So far none have been revealed. This suggests that the problem is not high-frequency. (Perhaps this could have been guessed from the fact that no one but me has reported it. )

[Trotterdam]

My suggestion would be to track the census score changes as reported by the issue-answering private command itself, and compare it against before-and-after data collected separately using the public API.

[Rocain Founder]

I now have another instance of the bug, with some of the details slightly different, and perhaps clarifying.

The nation is Cain BZ R08Au v0. Before I answered an issue, I used the API to get the government classification and the civil rights stat. The nation was Democratic Socialists. The civil rights stat was 36.11. I then answered issue #1186 via the API, and selected option 1 (or 0 as the API presents it). The returned XML from the issue answer contained neither a <RANK> item for a civil rights stat change nor a <RECLASSIFY> item for the government classification. However, when I used the API immediately thereafter, the classification had changed to Authoritarian Democracy, and the civil rights stat had dropped to 32.85. In other words, after answering the issue, the civil rights stat and the government classification changed without any sign of a change to either in the XML response to that issue.

At this moment, the game agrees that the government type and civil rights stat agree with what the API returned after I answered the issue. If you check the National Happenings, you will see, however, that the most recent government classification change recorded was in fact to Democratic Socialists (which the API recognized as the current classification before I answered the issue).

Due to what I saw with the previous instance, I suspect the stat and government classification may spontaneously change again overnight.

There can be little doubt that something is wrong here.

[Trotterdam]

I then answered issue #1186 via the API, and selected option 1 (or 0 as the API presents it).

Huh, weird. My data (both current and an old backup) shows that this option shouldn't have affected civil rights, and almost never does so (17 data points for the data as of last upload, and 187 data points in the old backup). However, my script did record the exact change you claim for your nation on this occasion, which will go online in a couple of hours, showing that you did your measurement correctly.

My conclusions:
1. When answering issues through the API, there is a definite discrepancy between the changes that the answering reported, and the changes that actually got saved.
2. Receiving (or not) the reclassification happenings line is based on the changes that were reported, but which classification your nation is thereafter actually shown as being is based on the changes that were saved.
3. I can't say with certainty which of the two change sets is actually the correct one that matches how the issue was intended to work, but I suspect that it's the reported changes, and not the saved ones.
4. The current assumption is that this is a bug with answering issues via the API, which incorrectly produces different results from when answering the same issue using the standard interface. However, it's possible that the bug can happen both ways but is simply very rare, though that strikes me as less likely.

Due to what I saw with the previous instance, I suspect the stat and government classification may spontaneously change again overnight.

No, I don't think it's overnight. It definitely looks like the bug happens at the moment of answering the issue. From what I can tell, your status then stays stable until the next issue answer, which may compound the bug and make things even more wrong (like Cain M Ca0 v0 being "reclassified" twice in a row and neither one actually sticking).

[Rocain Founder]

Due to what I saw with the previous instance, I suspect the stat and government classification may spontaneously change again overnight.

No, I don't think it's overnight. It definitely looks like the bug happens at the moment of answering the issue. From what I can tell, your status then stays stable until the next issue answer, which may compound the bug and make things even more wrong (like Cain M Ca0 v0 being "reclassified" twice in a row and neither one actually sticking).

Assuming I did not make an observation error (always possible), in the first case I reported, the civil rights stat and government type changed again after (as I recall) several days of inactivity, during which no issues were answered. You in fact confirmed that my observations of the stat at that time matched your own. I can't think of why a stat with no population dependency would change without provocation unless for some reason it was affected by the nightly update. I wonder if the nightly processing detected something amiss, and attempted to fix it. In any case, something happened to restore the government classification of Democratic Socialists after it had been repeatedly and apparently wrongly changed to Authoritarian Democracy. (Again, assuming my observations at the time were not incorrect.)

At any rate, I'm going to check on the nation tomorrow morning and post here whether my suspicions were confirmed or unfounded.

[Trotterdam]

In any case, something happened to restore the government classification of Democratic Socialists after it had been repeatedly and apparently wrongly changed to Authoritarian Democracy.

Did you ever actually see it be changed to Authoritarian Democracy, except in the happening line saying so?

Ah, hmm, I skimmed over this part of your report:

On July 14, around 2:30 EST, my script apparently was told, via the API, that the government type of Cain M Cao v0 was Democratic Socialists. It then answered issue #645, choosing option 3. According to the logs kept by my script, its civil rights stat dropped from 41.44 to 34.36, and the message ‘Cain M Cao v0 was reclassified from "Democratic Socialists" to "Authoritarian Democracy".’ was returned. An API inquiry confirmed that the government type was now Authoritarian Democracy. Other than the change-of-government message, I have no confirmation independent of my script of any of this.

Later that day, around 6:30 EST, I ran my script again. The API told the script that the government type was Authoritarian Democracy, but when I used the API to answer issue #458 with option 2, its civil rights stat dropped from 34.36 to 34.15, and I received the message ‘Cain M Cao v0 was reclassified from "Democratic Socialists" to "Authoritarian Democracy".’ once again. This is obviously wrong. Either the API was incorrect when it indicated the government type was already Authoritarian Democracy, or the change-of-government message was incorrect. The API call after answering the issue to retrieve the government type once more indicated it was Authoritarian Democracy.

If these reports are correct, then something even weirder is going on than what I thought. It seems to outright contradict the behavior you reported on Cain BZ R08Au v0, even though both are obviously bugged.

[Rocain Founder]

First, there was no change in the government classification of Cain BZ r08Au v0 this morning. So you were right about that, and I was wrong.

In any case, something happened to restore the government classification of Democratic Socialists after it had been repeatedly and apparently wrongly changed to Authoritarian Democracy.

Did you ever actually see it be changed to Authoritarian Democracy, except in the happening line saying so?

That "change" was also returned to me in the response to the issue choice by the API. Of course, the "change", like the national happening, reported a move from a government type that was wrong to the one it already had , at least according to the previous API call. My language in the sentence you quote was a little imprecise. The request for stats made immediately after response to the issue choice confirmed that the classification was still Authoritarian Democracy. I have no exact idea when it went back to being Democratic Socialists again (and my theory that it was a nightly update may well have been wrong), but I had later API confirmation that it had taken place.

I certainly agree with you that this whole thing is quite difficult to make sense of. But software bugs are often like that. Coding errors sometimes result in completely unreasonable and bizarre behavior.

[Rocain Founder]

I have another instance of this phenomenon. This morning, my puppet Cain G HiK Cmp v0 had a Civil Rights stat, as reported by the API, of 34.46, and a government type of Psychotic Dictatorship. It answered issue #975 with option 4 (3 to the API). The returned XML indicated no change to either the Civil Rights stat or to the government classification. However, an immediately following API query showed that the Civil Rights stat had risen to 35.44, and that the government type was now Corrupt Dictatorship. The game's display of that nation agrees with both the changed Civil Rights stat and the new government classification. No message about the government change was present in the National Happenings.

This incident is very similar to the previous incident.

I have paused answering issues for this nation for now, but I would like to resume as soon as I am told that doing so will not obscure any information which might be obtained on the nature of the problem from the nation's current state.

[Rocain Founder]

Earlier in this thread Trotterdam suggested that I should alter my script code to look for anomalies in the handling of the civil rights stat, specifically, unwarranted or lost changes in that stat. I finally got around to coding that, and I very much regret that it took me so long, as the results were starting. A typical full run of the script through my puppets processes around 70 of them, a significant fraction of whom, for any given run, dismiss their issue. After modifying the script and getting it debugged, there were 4 instances in the 70 nations where the civil rights stat was not shown as having been changed by an issue response, but then was changed after all when I got the stat again from NationStates! This comes to a frequency of about 1/18, but it's more than that due to the fact that there were the usual intermittent dismissals during the process.

For the record, here are the four nations and what happened:

Cain B I-Lb v0: Originally 22.85, after picking option 4(5) of issue #600, it rose to 24.00 without the issue response XML showing a change,

Cain B O-Sc v0: Originally 21.46, after picking option 3(4) of issue #1173, it fell to 21.42, without note in the XML issue response,

Cain G LoS Stm v0: Originally 20.15, after picking option 0(1) of issue #386, it fell to 20.08, without note in the XML issue response,

Cain A R-V Vg v1: Originally 19.38, after picking option 2(3) of issue #501, it rose to 19.71, without note in the XML issue response.

None of these changes were reflected in the government classification.

tl;dr: There appears to be a greater chance than 1 in 18 that the civil rights stat will be modified when answering an issue via the API in a way that is not reflected in the response data returned by the API. This is a bug.

But wait, there's more.

I updated my script to monitor all of the stat ids that I was processing. I was only able to process 16 more nations before I ran out of issues to answer. However, one time of the 16, the code was triggered, and in addition to the civil rights stat, the social conservatism and authoritarianism stats showed the same behavior. Here is the data:

Cain BZ R05Ce v0: Civil Rights went from 30.26 to 30.22, Social Conservatism went from 70.31 to 70.33 and Authoritarianism went from 862.46 to 862.71, after picking choice 0(1) of issue #496.

I don't see any reason that I shouldn't resume playing the nations which exhibited all this strangeness earlier - it's clear that I can generate new examples on demand, probably about 4 a day.

Players using the API who care about their results should be aware that at this time there seems to be an over 5% chance that things will go wrong.

I will continue running with this code - I can't rule out the possibility that other stats may be occasionally affected by this problem as well. I'll update the thread if I discover anything more of interest.

[Rocain Founder]

I have another instance now, where, in addition to the civil rights and social conservatism stats, the nudity, intelligence, ignorance, recreational drug use and patriotism stats were all changed without showing up in the issue response XML.

I'm not bothering to write out all the details by now - the point has been made. I've saved the XML file should anyone really wish to see all the numbers.

[Luna Amore]

Just commenting to say we're looking into this. Haven't found a root cause yet.

[[violet]]

I can't think of why a stat with no population dependency would change without provocation

Occasionally we change how scales are measured, and this happened recently with Civil Rights. How it normally works is the first time a nation logs in after this change, it gets adjusted - but possibly this is not happening (properly, or at all) for nations who never log in but only answer issues via the API.

[[violet]]

If the above hypothesis is correct, then these are the only currently active Cain_* nations who should experience this bug in the future, as they're the only ones who still have a now-obsolete "Freedom to Commit Crime" value:

Code: Select all

cain_a_r-o_au_v1
cain_bz_r04lb_v0
cain_ev_v1
cain_m_ca0_v1
cain_m_t06_v0
cain_o-n_wd_v1

[Rocain Founder]

Most of these nations do occasionally login to change their banners and other settings like issue frequency. (I suspect they all do, but I can't quite be 100% sure of that.) They have never answered an issue except via the API however. Some of them were created very recently, certainly after the recent Civil Rights change. (Cain A R-V Vg v1 is an example of one of those.) And there were several logins when these nations were created, though none of them involved answering issues.

[Rocain Founder]

Cain EF v1 (not on your list) had the bug about 2 hours ago. I don't know if that contradicts your hypothesis or not.

Tomorrow morning, when my nations' issues are refreshed, I'll run through them again and report on whether I saw the bug on any nation not on your list.

[[violet]]

I've found and fixed an issue that might be behind this -- some issues were still referring to "Freedom to Commit Crime" even after it had been removed from the game. Please do report the results of your further testing, thanks.

[Rocain Founder]

Five of the 6 nations on your list triggered the bug this morning. Cain M T06 v0 is a test nation, and I don't normally answer its issues, but after getting 4 hits on your list, I decided to do so today, and it was another hit. Only Cain M Ca0 v1 failed to trigger the bug. No nations not on your list exhibited the bug.

Data on each:

Cain BZ E04Lb v0: civil rights and social conservatism affected, issue #1252 option 2(3)

Cain EV v1: civil rights, social conservatism, intelligence, ignorance, ideological radicality and authoritarianism affected, issue #1374, option 1(2). This is the first time I've seen ideological radicality in the list.

Cain A R-O Au v1: civil rights, social conservatism, intelligence and ignorance affected, issue #818, option 0(1)

Cain O-N Wd v1: civil rights, social conservatism, intelligence, ignorance, ideological radicality and authoritarianism affected, issue #656 option 2(3)

Cain M T06 v0: civil rights and social conservatism affected, issue #285, option 1(2)

So the bug may be triggered by your extra crime freedom stat, which may have been added upon occasion by issues which still referenced it.

A couple questions: Do you know whether the problem is limited to the API? And do you know what stats might be modified incorrectly (assuming that the stats stored were wrong rather than that the actual error was leaving valid changes out of the issue response XML, which would be more benign)? One reason I ask is that if the data stored was wrong, Trotterdam would probably want to know what stats he needs to reset in his database once the problem is entirely solved. (I note that in one of the early instances of the problem, a change to the civil rights stat was noted by Trotterdam's software when no such change had previously been observed for that issue option, leading him to suspect that the data stored that time was not correct.)

[[violet]]

Thanks very much for that comprehensive testing! Hopefully you won't see this bug again with any more cain_* nations -- none of them have a "Freedom to Commit Crime" value any more.

A couple questions: Do you know whether the problem is limited to the API?

The problem originally appeared to be that some stat changes weren't being properly reported in the XML. That probably wasn't a bug at all, since the XML is only supposed to report stat changes due to answering that particular issue, and this stat change was due to a World Census rescaling. So I think we had this situation:

1. Nation sends API request to answer issue
2. Server loads nation
3. Server notices nation has obsolete "Freedom to Commit Crime" stat and deletes it
4. Nation answers issue
5. Server reports stat changes caused by #4, but not #3

So far, all this is fine and an unavoidable effect of making a stat obsolete. However, some issue choices were then adding back in a "Freedom to Commit Crime" stat. This meant two things: First, the nation could experience this same situation again in the future. Second, by creating a new stat that would only persist until the next login, the nation would effectively display stat changes that wouldn't seem to stick. This affected all nations, not just nations who answered issues via the API.

And do you know what stats might be modified incorrectly (assuming that the stats stored were wrong rather than that the actual error was leaving valid changes out of the issue response XML, which would be more benign)?

The primary one is Civil Rights, but maybe around half of all scales could have been affected, albeit on too small a scale to be detectable. Freedom is a fundamental stat that flows through to a wide range of World Census scales. It's not possible to answer definitively since it heavily depends on what the nation's personal freedom was before answering the issue, and of course the type of change executed by the issue choice. There would have been many cases when the bug was triggered but it wasn't visible even in Civil Rights, because the change was too small.

One reason I ask is that if the data stored was wrong, Trotterdam would probably want to know what stats he needs to reset in his database once the problem is entirely solved. (I note that in one of the early instances of the problem, a change to the civil rights stat was noted by Trotterdam's software when no such change had previously been observed for that issue option, leading him to suspect that the data stored that time was not correct.)

It's possible that any weird change to Civil Rights since June 12 has been caused by this, particularly if Civil Rights immediately reverted the next time the nation logged in.

There is no permanent damage to nations, since the bug involved adding and removing the same stat. The nation's underlying stats weren't affected, just the reported value of derived scales such as Civil Rights, and even then only in between immediately answering an issue that re-added "Freedom to Commit Crime" and the nation's next login.

It's also worth nothing that Editors are currently reviewing issues that I've removed "Freedom to Commit Crime" from, and may tweak other effects, particularly in cases where a change to FTCC was supposed to balance out a change to some other kind of freedom.

[Rocain Founder]

Wow! Thanks for all the detailed explanations!

I did in fact observe one case where there was no change visible to Civil Rights, but there were to Social Conservatism and Authoritarianism.

I have three non-Cain puppets (including myself), but I'm betting they do not still have the obsolete stat (if they ever did). So I'm glad to know this should be behind me. (Though I will keep my detection code active for a while, just in case.)

I have in only one case observed a change being reverted and, as I hadn't written any of the monitoring code yet at that point, I can't even be sure that's exactly what I saw, especially since I was concentrating on government classification at the time rather than civil rights.

Thanks again for all the effort it must have taken to work out what was going on so quickly.