Extreme update lag

[Alfonzo]

Hello,

Over the last few hours update has been running very strangely (at least major updates). This all started with small lag last major update but has now escalated to update being significantly halted this current update. Is it possible for this to be looked into?

Thank you.

[Domais]

Hello,

Over the last few hours update has been running very strangely (at least major updates). This all started with small lag last major update but has now escalated to update being significantly halted this current update. Is it possible for this to be looked into?

Thank you.

I can attest to this. Especially the triggering. It is messed up majorly.

[Tim-Opolis]

It would be really appreciated if Admin didn't ignore this one for days like the last thread.

I'm just gonna be blunt, the conclusion between myself and a few leaders across both sides of the faction is that it's not implausible that this is user generated. Personally, it's my main leading theory. It shouldn't be ignored that a supposed patch to the "DDOSing the site via Challenge" was mentioned, and yet after that we've seen things only get worse. Perhaps someone clued in and has been exploiting it worse?

For two majors straight, everything breaks right around/after The Rejected Realms updates, and starts running normally again after A Liberal Haven. Our two minute warning ran to almost seven minutes. Our estimated 12s trigger, which landed near-perfectly at 12s at minor update, ran seven minutes. This is well outside of usual server nonsense, someone is fucking with the game.

With all due respect to the Admin Team, please fix your game because a lot of us are starting to consider why we spend our money and time on something that can get constantly broken by bad faith actors this easily.

[Riet Grrrl]

Hey, we gathered literally a hundred people for a liberation, set a trigger, as one is wont to do in gameplay... only for update to slow down to the point of one nation every three seconds. For a sample:

08/01/2022, 5:33:29 GMT: Amagiri Ayato's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:28 GMT: Makasta being forced to merc's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:26 GMT: Bit of an L's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:25 GMT: Just as I was about to give up's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:23 GMT: Im good at defender's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:21 GMT: Miltray dictator of TL's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:19 GMT: Ohh ohhh ohhhhh's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:16 GMT: Something deep inside's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:14 GMT: Thank you mannal's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:11 GMT: Manunal recurtment i am sorry for you's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:08 GMT: Chadtopia of chadness's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:06 GMT: Suree im a native's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:03 GMT: Krunkoth's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:01 GMT: GCR soverty's influence in Russian Sfsr fell from "Hermit" to "Page".

A region of 700 nations took seven minutes to update (5:27:47 to 5:36:30).

To illustrate just how slow that is - earlier in update, a region of 11,000 nations took two and a half minutes (Chicken Overlords - 5:12:15 to 5:15:03), which is the typical speed.


Every other region seems to have been fine - issues started roughly when we issued our two minute warning (which was stretched out into 7 minutes), and continued until shortly after the jump. Update seemed to run fine after that.



If this is because the game cannot handle too many nations moving at once, or due to some kind of super-turbo-ultra-variance, that's concerning and please fix it. This could also be related.

[Reploid Productions]

Merging the duplicate threads. I've sent up the signal backstage, hopefully [v] will be able to take a look soon.

[Refuge Isle]

Supplemental information:

Lag started at approximately the start of TRR where nations suddenly took 140x extra to update. Until this point, update had been on schedule.

Pace of nations in our trigger region:



Pace of update now:


Nation updating speed went back to normal within 1-2 minutes of A Liberal Haven updated, where it remains normal up to the current time.

[Ever-Wandering Souls]

That's a lot of confidence in some mystery bad actor. Like, there's definitely some instability, and my sympathies for this not being a fair face-off at update for y'all to work on building your beachhead, but that's a lot of jumping to conclusions of specific malice against your lib, for what could have a dozen other technical explanations.

[Lord Dominator]

That's a lot of confidence in some mystery bad actor. Like, there's definitely some instability, and my sympathies for this not being a fair face-off at update for y'all to work on building your beachhead, but that's a lot of jumping to conclusions of specific malice against your lib, for what could have a dozen other technical explanations.

Will note this from yesterday major’s bit:

Should be fixed now. A player decided to rent some Amazon servers to send 50GB worth of bot requests to the Challenge page.

[Refuge Isle]

That's a lot of confidence in some mystery bad actor. Like, there's definitely some instability, and my sympathies for this not being a fair face-off at update for y'all to work on building your beachhead, but that's a lot of jumping to conclusions of specific malice against your lib, for what could have a dozen other technical explanations.

95% of this thread has thus far been posting nation updating times. I have a high degree of confidence in those timestamps.

Variance is one thing, a two minute time span becoming a twelve minute timespan when all other points on update appear normal is worth investigation.

[Ever-Wandering Souls]

That's a lot of confidence in some mystery bad actor. Like, there's definitely some instability, and my sympathies for this not being a fair face-off at update for y'all to work on building your beachhead, but that's a lot of jumping to conclusions of specific malice against your lib, for what could have a dozen other technical explanations.

95% of this thread has thus far been posting nation updating times. I have a high degree of confidence in those timestamps.

Variance is one thing, a two minute time span becoming a twelve minute timespan when all other points on update appear normal is worth investigation.

Certainly not saying it didn't happen, or that it shouldn't be investigated, more just that we're also performing a record number of actions per second for site history (with huge jumps and modern macros/tools), on two decade old spaghetti code, during an update that exists for an hour because 2 tin cans in Canada have to run at 100% capacity for that long to process it - so there's at least plausible other considerations, and we should be asking admin to investigate, before confidently going all in on that.

[Vincent Drake]

we're also performing a record number of actions per second for site history (with huge jumps and modern macros/tools)

We're really not. That huge stack of defenders refreshes the region once and hits a single button, that's 90 ish people making 2 clicks 6 minutes apart and not needing any scripts to do it. On the raider side, those pilers don't need to make any clicks, so it's just the handful of banners. This level of activity is well within what NS can handle. N-Day is extremely more intensive with tens of thousands of puppets being cycled constantly and nukes/shields flying en masse. Even then, NS only DDOSes due to container use - massive nuke strikes from people not using containers doesn't even produce a DDOS effect.

This sudden slowdown for a very specific period of time is not normal.

[Plasma Sage Ghestis]

Seems myself and potentially a few others had more specific issues pertaining to this topic, some activity feed and game behavior discrepancies. On the initial jump, I jumped with the nation I'm posting from here. At the time of the jump, this nation's activity feed contains my move into A Liberal Haven, a couple people endorsing me, an ejection and ban from the region by GonkDroid, a couple more people endorsing me, then my resignation and later subsequent re-admittance to the WA. In the interim there, I switched a couple times and tried jumping into the region again while the window to do so was still drug open by the massive site lag, then noticed that when I switched back onto this nation that it still behaves as though it's in A Liberal Haven and, from what I can tell, never actually moved to TRR. I confirmed that it can still endorse everyone endorsing Garchyland. My endorsements of these nations appear to still count towards their totals. Other people still appear to be able to endorse me in A Liberal Haven, and all of that activity pertaining to this nation in A Liberal Haven appears to be showing up in TRR's activity feed. It also appears as though this nation never updated in A Liberal Haven as a result of all that, so its influence hasn't changed and there's no corresponding message in the activity feed. The game still shows A Liberal Haven on the side bar as the region this nation is currently in, and I have a telegram from the website informing me of my ban and ejection from the region. Not really sure what to think of all that, but it's what it is. I can grab the relevant snips of activity feeds if they're helpful, otherwise I don't see the point of cluttering the thread when it's easy to click a nation and see the feed.

[Altmoras]

95% of this thread has thus far been posting nation updating times. I have a high degree of confidence in those timestamps.

Variance is one thing, a two minute time span becoming a twelve minute timespan when all other points on update appear normal is worth investigation.

Certainly not saying it didn't happen, or that it shouldn't be investigated, more just that we're also performing a record number of actions per second for site history (with huge jumps and modern macros/tools), on two decade old spaghetti code, during an update that exists for an hour because 2 tin cans in Canada have to run at 100% capacity for that long to process it - so there's at least plausible other considerations, and we should be asking admin to investigate, before confidently going all in on that.

This is a BS narrative, if normal player actions were what was causing site instability we would have seen that instability while the 98 defenders in Artificial Solar System were opening 97 pages each to cross endorse, not after our 2 minute warning when we stopped crossing. We should have seen an improvement in site speeds at the 2 minute warning if anything. The lag also started well before we jumped, it was not caused by our jump, the lag started when our 2 minute warning stretched to 6 minutes not when we gave the go command.

[Vando0sa]

Seems myself and potentially a few others had more specific issues pertaining to this topic, some activity feed and game behavior discrepancies. On the initial jump, I jumped with the nation I'm posting from here. At the time of the jump, this nation's activity feed contains my move into A Liberal Haven, a couple people endorsing me, an ejection and ban from the region by GonkDroid, a couple more people endorsing me, then my resignation and later subsequent re-admittance to the WA. In the interim there, I switched a couple times and tried jumping into the region again while the window to do so was still drug open by the massive site lag, then noticed that when I switched back onto this nation that it still behaves as though it's in A Liberal Haven and, from what I can tell, never actually moved to TRR. I confirmed that it can still endorse everyone endorsing Garchyland. My endorsements of these nations appear to still count towards their totals. Other people still appear to be able to endorse me in A Liberal Haven, and all of that activity pertaining to this nation in A Liberal Haven appears to be showing up in TRR's activity feed. It also appears as though this nation never updated in A Liberal Haven as a result of all that, so its influence hasn't changed and there's no corresponding message in the activity feed. The game still shows A Liberal Haven on the side bar as the region this nation is currently in, and I have a telegram from the website informing me of my ban and ejection from the region. Not really sure what to think of all that, but it's what it is. I can grab the relevant snips of activity feeds if they're helpful, otherwise I don't see the point of cluttering the thread when it's easy to click a nation and see the feed.

Oh hey! I just made another thread about this and another weird glitchy thing that happened.. we might have code gremlins..

We removed you from the banlist and banjected again so maybe it fixed it.. still weird though..

[Budgie Snugglers]

Using the order that nations updated in + the update happenings up to the 43rd minute of update, I made graphs - don't know if they are helpful. Data here.

[Eluvatar]

Thank you for bringing this to our attention and for some helpful graphs.

[Unibot III]

Those graphs are very illustrative. I can only remember a similar spike happening in back to back updates once in 2011, I forget the full context — Eluvatar would have the logs in #udl, searching for “Do not abort” (and “Hold the fucking line”).

Nothing else from my experience really matches this, and I think the possibility of manufactured lag or a DDOS attack should be taken seriously.

One way for the admins to know for sure is to move ALH to the back or front end of the update order. And see if the spikes continue, and when. I’m mostly joking.

[Plasma Sage Ghestis]

Seems myself and potentially a few others had more specific issues pertaining to this topic, some activity feed and game behavior discrepancies. On the initial jump, I jumped with the nation I'm posting from here. At the time of the jump, this nation's activity feed contains my move into A Liberal Haven, a couple people endorsing me, an ejection and ban from the region by GonkDroid, a couple more people endorsing me, then my resignation and later subsequent re-admittance to the WA. In the interim there, I switched a couple times and tried jumping into the region again while the window to do so was still drug open by the massive site lag, then noticed that when I switched back onto this nation that it still behaves as though it's in A Liberal Haven and, from what I can tell, never actually moved to TRR. I confirmed that it can still endorse everyone endorsing Garchyland. My endorsements of these nations appear to still count towards their totals. Other people still appear to be able to endorse me in A Liberal Haven, and all of that activity pertaining to this nation in A Liberal Haven appears to be showing up in TRR's activity feed. It also appears as though this nation never updated in A Liberal Haven as a result of all that, so its influence hasn't changed and there's no corresponding message in the activity feed. The game still shows A Liberal Haven on the side bar as the region this nation is currently in, and I have a telegram from the website informing me of my ban and ejection from the region. Not really sure what to think of all that, but it's what it is. I can grab the relevant snips of activity feeds if they're helpful, otherwise I don't see the point of cluttering the thread when it's easy to click a nation and see the feed.

Oh hey! I just made another thread about this and another weird glitchy thing that happened.. we might have code gremlins..

We removed you from the banlist and banjected again so maybe it fixed it.. still weird though..

Yep, looks like it worked on your second attempt there. I have the TG telling me I've been ejected and banned, I have TRR on my sidebar now, and I can no longer endorse nations in A Liberal Haven. Guess I'm definitely jumping again now

[[violet]]

Eluvatar and I have both been looking into this. Thanks again also for the details and charts above, which make it much easier to figure out what we should be looking at.

System load has remained normal since Jan 5, when we blocked a Challenge bot. For the week prior, and especially on Dec 30, 31st, and Jan 3, load from this bot caused update to run long.

Since then, update has more or less hit target length, but it did experience an unusual stall on Jan 7. At this point, I suspect the problem lies in the Activity log, which became backlogged attempting to record a very large number of entries. We haven't identified the root cause, but so far the behavior hasn't repeated.

Update is the single most load-intensive point we have, so it is sensitive to other load-generating events that would otherwise go unnoticed.

[Ever-Wandering Souls]

Eluvatar and I have both been looking into this. Thanks again also for the details and charts above, which make it much easier to figure out what we should be looking at.

System load has remained normal since Jan 5, when we blocked a Challenge bot. For the week prior, and especially on Dec 30, 31st, and Jan 3, load from this bot caused update to run long.

Since then, update has more or less hit target length, but it did experience an unusual stall on Jan 7. At this point, I suspect the problem lies in the Activity log, which became backlogged attempting to record a very large number of entries. We haven't identified the root cause, but so far the behavior hasn't repeated.

Update is the single most load-intensive point we have, so it is sensitive to other load-generating events that would otherwise go unnoticed.

Thank you for the update, the brief description of details, and for clarification that this was a technical issue with update hanging and not malicious activity.

[[violet]]

I've also contacted a few bot authors, because those have gotten out of control again. We have four in particular at the moment that are either illegal or broken (i.e. sending meaningless requests at maximum speed).

There's also something that identifies as"<Nation: NAME> Recruitment Tool." This looks legal and above board, but it's pretty inefficient, since it's being operated by multiple different nations, and they all look up the exact same Activity log data over and over every second or two. I don't know what this is or what it does, but it's a fair bit of load on that part of update.

[Roavin]

Thanks for the feedback, [v]! I do have some additional data to consider which might help you in your search:

Nakari's graphs only focus on the worst offender of the updates. However, of the four updates in which jumps into the region were performed, three were affected by this mysterious lag.

• Jan 7 at 12am EST (major): A nominal 14 second trigger became 66 seconds long. Update rate slowed down from over 30 to about 3-9 nations per second at the point of the trigger.
• Jan 7 at 12am EST (minor): This update did not show any irregularities.
• Jan 8 at 12am EST (major): This update is the bad one shown in Nakari's graphs
• Jan 8 at 12am EST (minor): A nominal 12 second trigger became 60 seconds long.

Collecting this data is a lot of work and it's very very difficult to do for minor updates (due to lack of a data dump), so here's additional data for that first major update. The start of the Rejected Realms updating was our estimated 2.5 minute warning for the target (by watching the activity feed until influence changes for TRR started). The nation Bladetress-22 was placed and estimated to update about 14 seconds before the target.

Relevant select happenings:

(1) 1/7/2022, 6:27:37 AM GMT+1: Bladetress-22's influence in The Rejected Realms rose from "Zero" to "Unproven".

Relevant data extracted from the region dump generated during that update:

• TRR began updating at 06:24:51 GMT+1
• Bernadina, the region immediately after TRR, began updating at 06:27:45 GMT+1
• Bladetress-22 was at position 5687/6146 in TRR
• 174 nations in 69 regions resided between TRR and A Liberal Haven
• A Liberal Haven (which I'll abbreviate as ALH going forward) began updating at 06:28:41 GMT+1

TRR's start occurred is 2 minutes and 46 seconds away from the designated trigger. This is within normal range (we estimated 2.5 minutes). However, the time from the trigger to the first ALH happening was 66 seconds, much more than the estimated 14 seconds. Doing a bit of math:

• 2 minutes and 46 seconds to update the first 5687 nations in TRR => 34.3 nations per second
• 66 seconds for the 632 nations from trigger to ALH => 9.6 nations per second
• 8 seconds for the 458 nations from trigger to end of TRR => 57.3 nations per second (actually faster!)
• 58 seconds for the 174 nations between TRR and ALH => 3 nations per second

Had the rate remained at 34.3 nations per second, the trigger length would have been 18 seconds, which is within normal parameters. The only abnormality I can see is that there were 28 new single-nation regions between TRR and ALH, but that seems an unlikely culprit because then the constant overhead for starting a region update would have to be nearly a second, which is clearly not the case.

----

Also, to provide some context as to why some individuals think this may be foul play: This is a kind of spike that we haven't seen before, and it only happens right around the time a jump into an occupied region takes place. Even more suspiciously, of the four successive updates where a jump takes place, one of them is not affected, that being the weekday minor update which are traditionally the ones were the least amount of gameplayers are available (and, if this is a malicious actor, they would have to be a R/D gameplayer due to the timing). I hope it's just server gremlins, but I can't fault suspicions that it's not, and at this point we have neither proof that it is nor that it isn't, and should probably drop both speculations and undue celebrations until [v] and Elu know more.

[Eluvatar]

At this point, I suspect the problem lies in the Activity log, which became backlogged attempting to record a very large number of entries. We haven't identified the root cause, but so far the behavior hasn't repeated.

I have some hypotheses for which data on happenings that happened more than 6 days ago would be helpful. As a gameplayer, I used to log all happenings for purposes of analyzing update, game mechanics, etc. If any players happen to have been doing that recently, I would much appreciate the submission of a log of happenings since the start of the year over GHR. If nobody has it, that's okay, there are other avenues for analysis they just might take longer.

[[violet]]

Also, to provide some context as to why some individuals think this may be foul play: This is a kind of spike that we haven't seen before, and it only happens right around the time a jump into an occupied region takes place. Even more suspiciously, of the four successive updates where a jump takes place, one of them is not affected, that being the weekday minor update which are traditionally the ones were the least amount of gameplayers are available (and, if this is a malicious actor, they would have to be a R/D gameplayer due to the timing). I hope it's just server gremlins, but I can't fault suspicions that it's not, and at this point we have neither proof that it is nor that it isn't, and should probably drop both speculations and undue celebrations until [v] and Elu know more.

There was a code change on December 2, 2021 that increased the workload caused by a region move. (The process now has to load the target region twice, not just once.) Doesn't line up particularly well with the issue here, but it's the only relevant system change I can think of. Potentially it might create noticeable slowdown when large numbers of nations are moving into the same region at the same time.

[Roavin]

There was a code change on December 2, 2021 that increased the workload caused by a region move. (The process now has to load the target region twice, not just once.) Doesn't line up particularly well with the issue here, but it's the only relevant system change I can think of. Potentially it might create noticeable slowdown when large numbers of nations are moving into the same region at the same time.

Yeah, it doesn't quite line up for two reasons that I can think of. First, the slowdown happened before all the nations started moving. Second, then we would have expected to see similar things happening in China on December 20, where the numbers were almost as large.