Extreme update lag

[Sedgistan]

Now that it looks like we have a conclusion to the situation, I feel this needs to be addressed. Tim, this post of yours has aged really badly, and it really goes to show why you need to change your attitude, particularly when posting in Technical:

It would be really appreciated if Admin didn't ignore this one for days like the last thread.

I'm just gonna be blunt, the conclusion between myself and a few leaders across both sides of the faction is that it's not implausible that this is user generated. Personally, it's my main leading theory. It shouldn't be ignored that a supposed patch to the "DDOSing the site via Challenge" was mentioned, and yet after that we've seen things only get worse. Perhaps someone clued in and has been exploiting it worse?

For two majors straight, everything breaks right around/after The Rejected Realms updates, and starts running normally again after A Liberal Haven. Our two minute warning ran to almost seven minutes. Our estimated 12s trigger, which landed near-perfectly at 12s at minor update, ran seven minutes. This is well outside of usual server nonsense, someone is fucking with the game.

With all due respect to the Admin Team, please fix your game because a lot of us are starting to consider why we spend our money and time on something that can get constantly broken by bad faith actors this easily.

Technical is a forum for people to work together constructively to improve the game. There's no need for that level of hostility - it puts people (including Admin) off engaging here, and that helps no-one. Likewise, the attitude towards the raider side - accusing them of deliberately DDOSing the site - is unacceptable. I hope you can bring yourself to apologise for that one.

[Astrobolt]

Now that it looks like we have a conclusion to the situation, I feel this needs to be addressed. Tim, this post of yours has aged really badly, […]

Is this confirmation that Storm was responsible for the issues on the 7th and the 8th? Because if so, that wasn’t explicitly stated before.

[Kanta Hame]

At least that makes sense and sounds legit option. And It can be even tested by / with mods for example in any of the feeders / sinkers during pre determined update and see what happens to either confirm or count out.

[Eluvatar]

Now that it looks like we have a conclusion to the situation, I feel this needs to be addressed. Tim, this post of yours has aged really badly, […]

Is this confirmation that Storm was responsible for the issues on the 7th and the 8th? Because if so, that wasn’t explicitly stated before.

No, and saying we have a conclusion is probably overstating things a little bit. Storm overload is worth investigating but has not been established as causing the extreme lag originally reported. But I wouldn't disagree with what Sedgistan is saying about tone!

[Kanaia]

Now that it looks like we have a conclusion to the situation, I feel this needs to be addressed. Tim, this post of yours has aged really badly, and it really goes to show why you need to change your attitude, particularly when posting in Technical:

It would be really appreciated if Admin didn't ignore this one for days like the last thread.

I'm just gonna be blunt, the conclusion between myself and a few leaders across both sides of the faction is that it's not implausible that this is user generated. Personally, it's my main leading theory. It shouldn't be ignored that a supposed patch to the "DDOSing the site via Challenge" was mentioned, and yet after that we've seen things only get worse. Perhaps someone clued in and has been exploiting it worse?

For two majors straight, everything breaks right around/after The Rejected Realms updates, and starts running normally again after A Liberal Haven. Our two minute warning ran to almost seven minutes. Our estimated 12s trigger, which landed near-perfectly at 12s at minor update, ran seven minutes. This is well outside of usual server nonsense, someone is fucking with the game.

With all due respect to the Admin Team, please fix your game because a lot of us are starting to consider why we spend our money and time on something that can get constantly broken by bad faith actors this easily.

Technical is a forum for people to work together constructively to improve the game. There's no need for that level of hostility - it puts people (including Admin) off engaging here, and that helps no-one. Likewise, the attitude towards the raider side - accusing them of deliberately DDOSing the site - is unacceptable. I hope you can bring yourself to apologise for that one.

Ya'll hear that swooshing noise? Yeah, that's the sound of Sedge attempting to sweep this under the rug, and then trying to shame Tim.

[Reploid Productions]

Ya'll hear that swooshing noise? Yeah, that's the sound of Sedge attempting to sweep this under the rug, and then trying to shame Tim.

Quick reminder that Bad Faith can be applied as needed to Technical threads, and this thread certainly falls in that category of "non-constructive, points-scoring posts are not welcome." Continued posts made in blatant bad faith can and will be moved to the evidence locker per [violet]'s directives on bad faith posting.

The techies are still investigating and have provided updates when and where they can. They don't want to conclusively say "X was the problem", as they do not yet know for absolute certain that X was the problem and don't want to give potentially inaccurate information. At this time the Storm tool is a likely suspect or at least potential contributing factor, but it is still unknown if it is the only factor, whether there are other issues at play, and so on. "Mods/admins sweeping it under the rug!" potshots contribute precisely nothing of value to an otherwise reasonably productive discussion in trying to ascertain what happened.

~Evil Forum Empress Rep Prod the Ninja Admin
~She who wields the Banhammer; master of the mighty moderation no-dachi Kiritateru Teikoku

[Zizou]

This is only speculation, but considering that the lag on 7-8 January started after the 2-minute warning (when people don't generally continue to cross) and indeed, after the "go" trigger, I don't see why people would have been using a cross feature at that time.

The problem is that the API requests don't stop when people stop crossing. The API requests are made in the background while the person is crossing. If something like I described above happened, the requests would not stop until the person did something to make them stop, such as closing their browser or uninstalling the extension.

After a bit of discussion and some time to sleep on the issue, I'd like to elaborate on this a bit. If memory serves me correctly, you guys sent Garchy into the region a few minutes before the rest of the updater force to ensure that he would update in ALH. That means that if someone was crossing off of Garchy, they probably would have switched to crossing off of another nation to finish up their crossing. That is pretty much exactly something that would cause Storm to experience the buggy behavior that I described earlier.

In addition to that, Garchy would have probably had more endorsements than anyone else in the cross. This means that anyone who switched off of crossing off him to another nation is liable to have triggered the bug that starts spamming the server with useless requests. And as explained earlier, because the stop condition for these requests is broken, any press of the C key will increase the rate at which these requests are sent until they are endlessly being sent at a speed which is many times greater than the rate limit.

[Ever-Wandering Souls]

The problem is that the API requests don't stop when people stop crossing. The API requests are made in the background while the person is crossing. If something like I described above happened, the requests would not stop until the person did something to make them stop, such as closing their browser or uninstalling the extension.

After a bit of discussion and some time to sleep on the issue, I'd like to elaborate on this a bit. If memory serves me correctly, you guys sent Garchy into the region a few minutes before the rest of the updater force to ensure that he would update in ALH. That means that if someone was crossing off of Garchy, they probably would have switched to crossing off of another nation to finish up their crossing. That is pretty much exactly something that would cause Storm to experience the buggy behavior that I described earlier.

In addition to that, Garchy would have probably had more endorsements than anyone else in the cross. This means that anyone who switched off of crossing off him to another nation is liable to have triggered the bug that starts spamming the server with useless requests. And as explained earlier, because the stop condition for these requests is broken, any press of the C key will increase the rate at which these requests are sent until they are endlessly being sent at a speed which is many times greater than the rate limit.

If Ziz's theory and assement of the bug conditions are accurate, one supporting bit of evidence could be seeing if anyone *does* appear to have kept crossing after Garchy left, despite that not being something people "generally" continue to do. Does anyone have the ASS activity feed logs necessary to check for that?

I know the question has been raised as well "if this is the case, why didn't we see similar slowdowns on earlier libs where this tool was likely used?" Again if Ziz's theories are accurate, the explanation could be either "the point didn't leave early so there wasn't a reason to start crossing on someone else" or, per the comment that typical procedure would have crossing stop then, "normally no one tries to keep crossing then but some noobs along for this lib did."

I think, once more /assuming/ ziz has a good read on the bug, this lines up fairly well with "why did this start around the 2 minute mark and not earlier in the cross?"

[Roavin]

Nice catch, Zizou - that's certainly a compelling hypothesis. I'd say it's absolutely plausible, but I wouldn't say that it's that likely (unfortunately).

The problem is that the API requests don't stop when people stop crossing. The API requests are made in the background while the person is crossing. If something like I described above happened, the requests would not stop until the person did something to make them stop, such as closing their browser or uninstalling the extension.

I didn't try Storm myself but from perusing the code, I see a third condition: Starting a new cross-endorse process on another target nation and/or with another user nation. In that scenario, the undefined bug remains, but all the parallel API calls cease at that point.

If memory serves me correctly, you guys sent Garchy into the region a few minutes before the rest of the updater force to ensure that he would update in ALH.

Yes, Garchyland was always sent back to ALH shortly after the nominal 5 minute warning.

That means that if someone was crossing off of Garchy, they probably would have switched to crossing off of another nation to finish up their crossing. That is pretty much exactly something that would cause Storm to experience the buggy behavior that I described earlier.

Does storm show when the target nation is no longer in the region and can't be endorsed?

The Storm users we know of from those updates aren't experienced enough in R/D to think to switch to another nation to top off their cross, plus all of them checked into Libcord's staging area relatively early so they wouldn't be compelled to panic-cross at the last moment. Furthermore, we never ordered other cross points (sometimes we do, but we didn't this time, I double checked), and even once in region people were ordered to continue crossing on Garchy.

That being said, I'll ask around to see if one of the Storm users present can remember their usage pattern.

If Ziz's theory and assement of the bug conditions are accurate, one supporting bit of evidence could be seeing if anyone *does* appear to have kept crossing after Garchy left, despite that not being something people "generally" continue to do. Does anyone have the ASS activity feed logs necessary to check for that?

The Activity log doesn't show "failed" endorsements, so we (non-admins) can't check that.

I know the question has been raised as well "if this is the case, why didn't we see similar slowdowns on earlier libs where this tool was likely used?" Again if Ziz's theories are accurate, the explanation could be either "the point didn't leave early so there wasn't a reason to start crossing on someone else" or, per the comment that typical procedure would have crossing stop then, "normally no one tries to keep crossing then but some noobs along for this lib did."

On the first point ("the point didn't leave early"), we've been using that tactic a considerable amount, including on updates where according to our reports probable-but-not-yet-confirmed Storm user were present.

On the second point, I can't say if this is a factor because I don't know how Storm reacts when crossing on a nation that, in the middle of the crossing process, leaves the region. If this isn't user-visible, then it's unlikely to think for most if not all of the probable Storm users that they'd think to switch to another nation to finish a cross off when this wasn't ordered and they didn't join last-minute.

---

So, TL;DR: Based on the above, I'd say the hypothesis is absolutely plausible but somewhat unlikely. That being said, I can imagine Admins being able to relatively easily verify or falsify this particular scenario based on server logs. Elu?

[Vando0sa]

Maybe some trial and error checking.. do another lib on Saturday major using storm to see if it happens again and if it does do another one Sunday major without it to see of things go different?

[Kanta Hame]

I am not familiar with it and just asking if any our members does. Its not in our helpful tools list so would have required someone to find and used individually. I asked if anyone has it but thus far no replies.

I have no clue how new this tool is but if existed and used since November its been used for all +100 peeps jumps with out issues ergo spamming C too much with too many players is not common issue until AHL. I saw list that contained peeps showing up for all borked updates and none of them were known storm users (not 100% is this verified + are not my members so take this bit with grain of salt).

I have no clue how many time one person would need to miss use it. But if one person it seemingly be that unlike any of previous 3-4 big ops, every jump same mist take would have needed to be made by different person. Possible but un likely. Unless of course one of those who participated all updates uses it and have spammed C due lack of patience or to make sure its pressed with out knowing it causes issues. But it would require one those in short list who made all attempts in bonked updates to have STORM after all and if so its more likely option. Different people doing repeatedly same mistake individually on next updates is statistically low odds option.

If multiple people are required to cause effect, then such thing happening via accident is even less likely as it starts to require even more people doing same mistake on different updates. Possible tho.

Also if known issue by some, could be easily be intentionally bork update too.

Regardless, first would need to define how much one can alone bork update with or, can they or would it need multiple people to do so. This can even tried multiple time on update to see how many times one person would need to click C till it has affect or how many people would need to do it by double clicking until there is effect. And if it matches what we experienced would match and data gathered helping identify witch users caused it.

[Roavin]

More data!

I've put together the 4 updates in question as well as the 2 updates from the China liberation in December, added in which of the known 5 Storm users (anonymized, referred to as 1-5) were present, as well as how that update faired (based in part on contemporary posts in Libcord).

Update	Storm users	Result
12/19 minor (China)	1, 2, 4	trigger went long (23 seconds)
12/20 major (China)	none	no problems
1/7 major	4	trigger went long (>60 seconds)
1/7 minor	1, 2	no problems
1/8 major	3, 4	very broken (7 minutes)
1/8 minor	3	trigger went long (>60 seconds)

This definitely gives credence to Zizou's Storm hypothesis, with particularly the usage patterns by users 3 and 4. I'll reach out to them to ask about how they used it.

EDIT - I got responses.
User 3 noted that they had used Storm to endotart on their main about a day before they attended the liberation, and didn't restart Firefox in the meantime. That would certainly trigger the bug.
User 4 noted that they usually keep their browser closed between using it, making it unlikely that they would trigger the bug. Furthermore, for the last ALH jump, the feature didn't work at all for them so they ended up manually cross-endorsing.

[Ever-Wandering Souls]

This all seems pretty plausible.
-The timing during update (getting bad closer to the warning than the jump) generally makes sense
-The nature of how the bug ramps up (and [v] having seen irl how quickly it can ramp up to 100% of Normal Site Traffic) makes sense
-The fact it continues until specific things end it that are more likely to happen post-update (i.e. closing windows), thus the issue ends shortly after ALH updates, makes sense
-The idea of an extra 100% site traffic, or more (given [v] said she observed 130/s as approx 100%, which is almost 500k/hr, and Elu has said closer to 100k/hr is more typical) via the API causing or majorly contributing to a slowdown of update as the system struggles with the load makes sense
-The question of "why haven't we seen this more often" is a valid one, but it looks like maybe we *did* see it at least once during China per Roavin's most recent post, and if we're running with the assumption that one or two specific people are operating this tool in a way that triggers the bug (whether those you numbered or others), it mostly makes sense that its presence is intermittent. Also, the nature of the bug potentially makes sense with the effect being worse the bigger the lib is/the more people one is trying to cross. Or if two people are triggering it, and both are present vs one. Responses aside, that uh, sure *looks* like a pattern of it being bad when either of users 3 or 4 are present, and worst when both are. Small sample size though!

As others have said, it would be wonderful if Admin had any data to support this, or could give us information to help support this (i.e. if you have such records, when did the API get inundated with these calls on the relevant nights? Can we match that to any other events using Discord, activity, or other records?), but in general....I've been saying since Ziz first dropped this hypothesis that I've been like 80-85 percent confident in it and I've probably breached 90% by now personally =P

[Kanta Hame]

Then needs to determine how badly one or two players can bonk update with it and what it requires until it puts update to crawl.

[Alfonzo]

I'm glad we (mostly) nipped this one in the bud

Even better, it seems Storm has been fixed recently to stop this. Thank you Admins!

[Zizou]

User 4 noted that they usually keep their browser closed between using it, making it unlikely that they would trigger the bug. Furthermore, for the last ALH jump, the feature didn't work at all for them so they ended up manually cross-endorsing.

Nice, thanks for gathering the relevant data. One thing I'd like to note though is that it's still possible that User 4 triggered the bug as well. Closing their browser in between usages would certainly help, since it would effectively "reset" the script if it was bugged. However, you note that on ALH they tried to use it, and found that it did not work and ended up manually cross endorsing. That still leaves somewhat of a possibility that they ended up triggering the bug.

Sending requests to the API and the actual cross endorsing are done somewhat independently of one another, since the requests to the API are to avoid wasting time trying to endorse nations which the user has already endorsed. Therefore, it's possible for one of these things to fail and the other to still perform properly. This is why even if the undefined bug is triggered, the actual cross endorsing functionality still continues to work. Likewise, if for some reason the actual cross endorse functionality is not working (say for example if your nation isn't in the WA, or if you're in a different region than the nation you're trying to cross off of), the script will continue to issue API requests in the background and it's still possible to trigger the bug.

So it's still possible to have triggered the bug if said user had tried to cross endorse off of a nation, it failed, tried cross endorsing off of another nation to try and fix the issue, failed again, and then after that they gave up and started cross endorsing manually. Although confirming this would require a more in depth recollection of what exactly User 4 tried before switching to manually cross endorsing.