NATION

PASSWORD

Extreme update lag

Bug reports, general help, ideas for improvements, and questions about how things are meant to work.
User avatar
Alfonzo
Envoy
 
Posts: 218
Founded: Dec 27, 2020
Democratic Socialists

Issues with Update

Postby Alfonzo » Fri Jan 07, 2022 10:40 pm

Hello,

Over the last few hours update has been running very strangely (at least major updates). This all started with small lag last major update but has now escalated to update being significantly halted this current update. Is it possible for this to be looked into?

Thank you.
✯ ✯ ✯ In War, Victory. In Peace, Vigilance. In Death, Sacrifice: TGW ✯ ✯ ✯
Made ya look!

User avatar
Domais
Attaché
 
Posts: 89
Founded: Sep 15, 2018
Inoffensive Centrist Democracy

Postby Domais » Fri Jan 07, 2022 10:42 pm

Alfonzo wrote:Hello,

Over the last few hours update has been running very strangely (at least major updates). This all started with small lag last major update but has now escalated to update being significantly halted this current update. Is it possible for this to be looked into?

Thank you.

I can attest to this. Especially the triggering. It is messed up majorly.
Prime Minister of Lazarus!
Delegate Emeritus of Warzones, Australia, Asia, Airspace and Africa
Voted Best Delegate of Warzone Airspace for the year 2019!

User avatar
Tim-Opolis
Negotiator
 
Posts: 6068
Founded: Feb 17, 2010
Iron Fist Consumerists

Postby Tim-Opolis » Fri Jan 07, 2022 10:50 pm

It would be really appreciated if Admin didn't ignore this one for days like the last thread.

I'm just gonna be blunt, the conclusion between myself and a few leaders across both sides of the faction is that it's not implausible that this is user generated. Personally, it's my main leading theory. It shouldn't be ignored that a supposed patch to the "DDOSing the site via Challenge" was mentioned, and yet after that we've seen things only get worse. Perhaps someone clued in and has been exploiting it worse?

For two majors straight, everything breaks right around/after The Rejected Realms updates, and starts running normally again after A Liberal Haven. Our two minute warning ran to almost seven minutes. Our estimated 12s trigger, which landed near-perfectly at 12s at minor update, ran seven minutes. This is well outside of usual server nonsense, someone is fucking with the game.

With all due respect to the Admin Team, please fix your game because a lot of us are starting to consider why we spend our money and time on something that can get constantly broken by bad faith actors this easily.
Last edited by Tim-Opolis on Fri Jan 07, 2022 11:38 pm, edited 1 time in total.
Want to be a hero? Join The Grey Wardens - Help Us Save Nationstates
You can trust me, I'm the Most Influential Player and Gameplayer of The Year in 2021.

Author of SC#74, SC #203, SC #222, and SC #238 | Co-Author of SC#191
Founder of Spiritus | Three-Time Delegate of Osiris | Pharaoh of the Islamic Republics of Iran | Hero of Greece
<Koth - 06/30/2020> I mean as far as GPers go, Tim is one of the most iconic

User avatar
Riet Grrrl
Civilian
 
Posts: 1
Founded: Apr 02, 2019
Democratic Socialists

Extreme update lag

Postby Riet Grrrl » Fri Jan 07, 2022 10:51 pm

Hey, we gathered literally a hundred people for a liberation, set a trigger, as one is wont to do in gameplay... only for update to slow down to the point of one nation every three seconds. For a sample:

08/01/2022, 5:33:29 GMT: Amagiri Ayato's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:28 GMT: Makasta being forced to merc's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:26 GMT: Bit of an L's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:25 GMT: Just as I was about to give up's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:23 GMT: Im good at defender's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:21 GMT: Miltray dictator of TL's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:19 GMT: Ohh ohhh ohhhhh's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:16 GMT: Something deep inside's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:14 GMT: Thank you mannal's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:11 GMT: Manunal recurtment i am sorry for you's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:08 GMT: Chadtopia of chadness's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:06 GMT: Suree im a native's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:03 GMT: Krunkoth's influence in Russian Sfsr fell from "Hermit" to "Page".
08/01/2022, 5:33:01 GMT: GCR soverty's influence in Russian Sfsr fell from "Hermit" to "Page".


A region of 700 nations took seven minutes to update (5:27:47 to 5:36:30).

To illustrate just how slow that is - earlier in update, a region of 11,000 nations took two and a half minutes (Chicken Overlords - 5:12:15 to 5:15:03), which is the typical speed.


Every other region seems to have been fine - issues started roughly when we issued our two minute warning (which was stretched out into 7 minutes), and continued until shortly after the jump. Update seemed to run fine after that.



If this is because the game cannot handle too many nations moving at once, or due to some kind of super-turbo-ultra-variance, that's concerning and please fix it. This could also be related.

User avatar
Reploid Productions
Forum Admin
 
Posts: 29019
Founded: Antiquity
Democratic Socialists

Postby Reploid Productions » Fri Jan 07, 2022 10:54 pm

Merging the duplicate threads. I've sent up the signal backstage, hopefully [v] will be able to take a look soon.
Forum mod since May 8, 2003 -- Game mod since May 19, 2003 -- Nation turned 18 on March 23, 2021!
Sunset's DoGA FAQ - For those using DoGA to make their NS military and such.
One Stop Rules Shop -- Reppy's Sig Workshop -- Getting Help Page
[violet] wrote:Maybe we could power our new search engine from the sexual tension between you two.
Char Aznable/Giant Meteor 2024! - Forcing humanity to move into space and progress whether we goddamn want to or not!

User avatar
Refuge Isle
Ambassador
 
Posts: 1016
Founded: Dec 14, 2018
Left-wing Utopia

Postby Refuge Isle » Fri Jan 07, 2022 10:57 pm

Supplemental information:

Lag started at approximately the start of TRR where nations suddenly took 140x extra to update. Until this point, update had been on schedule.

Pace of nations in our trigger region:
Image


Pace of update now:
Image

Nation updating speed went back to normal within 1-2 minutes of A Liberal Haven updated, where it remains normal up to the current time.

Ikania wrote:My brother in Christ, please stop posting.

User avatar
Ever-Wandering Souls
Negotiator
 
Posts: 6748
Founded: Jan 01, 2014
Father Knows Best State

Postby Ever-Wandering Souls » Fri Jan 07, 2022 10:57 pm

That's a lot of confidence in some mystery bad actor. Like, there's definitely some instability, and my sympathies for this not being a fair face-off at update for y'all to work on building your beachhead, but that's a lot of jumping to conclusions of specific malice against your lib, for what could have a dozen other technical explanations.
Proud Raider; General of The Black Hawks, Ret.
TG me anytime; I'm always happy to talk about anything!

The Alicorns (Equestria) wrote:Let them stay, no need to badmouth them...From our view a bunch of nations just came in, seized the delegate position, and changed a few superficial things...we play NationStates differently...there's really no reason for us to be butthurt.
http://www.nationstates.net/page=rmb/postid=8944227
http://www.nationstates.net/page=rmb/postid=8951258

Misley wrote:
Hobbesistan wrote:Don't think I understand the question.
The color or what?..

Jesus, Hobbes, it's 2015. You can't just call someone "the color".

Reploid Productions wrote:Raiders are endlessly creative

How Do I Telegram API?

Omnis delenda est.

User avatar
Lord Dominator
Powerbroker
 
Posts: 8221
Founded: Dec 22, 2016
Compulsory Consumerist State

Postby Lord Dominator » Fri Jan 07, 2022 11:02 pm

Ever-Wandering Souls wrote:That's a lot of confidence in some mystery bad actor. Like, there's definitely some instability, and my sympathies for this not being a fair face-off at update for y'all to work on building your beachhead, but that's a lot of jumping to conclusions of specific malice against your lib, for what could have a dozen other technical explanations.

Will note this from yesterday major’s bit:
[violet] wrote:Should be fixed now. A player decided to rent some Amazon servers to send 50GB worth of bot requests to the Challenge page.

User avatar
Refuge Isle
Ambassador
 
Posts: 1016
Founded: Dec 14, 2018
Left-wing Utopia

Postby Refuge Isle » Fri Jan 07, 2022 11:05 pm

Ever-Wandering Souls wrote:That's a lot of confidence in some mystery bad actor. Like, there's definitely some instability, and my sympathies for this not being a fair face-off at update for y'all to work on building your beachhead, but that's a lot of jumping to conclusions of specific malice against your lib, for what could have a dozen other technical explanations.

95% of this thread has thus far been posting nation updating times. I have a high degree of confidence in those timestamps.

Variance is one thing, a two minute time span becoming a twelve minute timespan when all other points on update appear normal is worth investigation.

Ikania wrote:My brother in Christ, please stop posting.

User avatar
Ever-Wandering Souls
Negotiator
 
Posts: 6748
Founded: Jan 01, 2014
Father Knows Best State

Postby Ever-Wandering Souls » Fri Jan 07, 2022 11:18 pm

Refuge Isle wrote:
Ever-Wandering Souls wrote:That's a lot of confidence in some mystery bad actor. Like, there's definitely some instability, and my sympathies for this not being a fair face-off at update for y'all to work on building your beachhead, but that's a lot of jumping to conclusions of specific malice against your lib, for what could have a dozen other technical explanations.

95% of this thread has thus far been posting nation updating times. I have a high degree of confidence in those timestamps.

Variance is one thing, a two minute time span becoming a twelve minute timespan when all other points on update appear normal is worth investigation.


Certainly not saying it didn't happen, or that it shouldn't be investigated, more just that we're also performing a record number of actions per second for site history (with huge jumps and modern macros/tools), on two decade old spaghetti code, during an update that exists for an hour because 2 tin cans in Canada have to run at 100% capacity for that long to process it - so there's at least plausible other considerations, and we should be asking admin to investigate, before confidently going all in on that.
Last edited by Ever-Wandering Souls on Fri Jan 07, 2022 11:18 pm, edited 1 time in total.
Proud Raider; General of The Black Hawks, Ret.
TG me anytime; I'm always happy to talk about anything!

The Alicorns (Equestria) wrote:Let them stay, no need to badmouth them...From our view a bunch of nations just came in, seized the delegate position, and changed a few superficial things...we play NationStates differently...there's really no reason for us to be butthurt.
http://www.nationstates.net/page=rmb/postid=8944227
http://www.nationstates.net/page=rmb/postid=8951258

Misley wrote:
Hobbesistan wrote:Don't think I understand the question.
The color or what?..

Jesus, Hobbes, it's 2015. You can't just call someone "the color".

Reploid Productions wrote:Raiders are endlessly creative

How Do I Telegram API?

Omnis delenda est.

User avatar
Vincent Drake
Envoy
 
Posts: 346
Founded: Dec 08, 2016
Inoffensive Centrist Democracy

Postby Vincent Drake » Fri Jan 07, 2022 11:27 pm

we're also performing a record number of actions per second for site history (with huge jumps and modern macros/tools)


We're really not. That huge stack of defenders refreshes the region once and hits a single button, that's 90 ish people making 2 clicks 6 minutes apart and not needing any scripts to do it. On the raider side, those pilers don't need to make any clicks, so it's just the handful of banners. This level of activity is well within what NS can handle. N-Day is extremely more intensive with tens of thousands of puppets being cycled constantly and nukes/shields flying en masse. Even then, NS only DDOSes due to container use - massive nuke strikes from people not using containers doesn't even produce a DDOS effect.

This sudden slowdown for a very specific period of time is not normal.
Commander in The Order of the Grey Wardens
Founder of European Union

Need to talk? Vincent Drake#3952

User avatar
Plasma Sage Ghestis
Political Columnist
 
Posts: 2
Founded: Nov 28, 2021
Inoffensive Centrist Democracy

Postby Plasma Sage Ghestis » Fri Jan 07, 2022 11:41 pm

Seems myself and potentially a few others had more specific issues pertaining to this topic, some activity feed and game behavior discrepancies. On the initial jump, I jumped with the nation I'm posting from here. At the time of the jump, this nation's activity feed contains my move into A Liberal Haven, a couple people endorsing me, an ejection and ban from the region by GonkDroid, a couple more people endorsing me, then my resignation and later subsequent re-admittance to the WA. In the interim there, I switched a couple times and tried jumping into the region again while the window to do so was still drug open by the massive site lag, then noticed that when I switched back onto this nation that it still behaves as though it's in A Liberal Haven and, from what I can tell, never actually moved to TRR. I confirmed that it can still endorse everyone endorsing Garchyland. My endorsements of these nations appear to still count towards their totals. Other people still appear to be able to endorse me in A Liberal Haven, and all of that activity pertaining to this nation in A Liberal Haven appears to be showing up in TRR's activity feed. It also appears as though this nation never updated in A Liberal Haven as a result of all that, so its influence hasn't changed and there's no corresponding message in the activity feed. The game still shows A Liberal Haven on the side bar as the region this nation is currently in, and I have a telegram from the website informing me of my ban and ejection from the region. Not really sure what to think of all that, but it's what it is. I can grab the relevant snips of activity feeds if they're helpful, otherwise I don't see the point of cluttering the thread when it's easy to click a nation and see the feed.

User avatar
Altmoras
Diplomat
 
Posts: 819
Founded: Jan 25, 2012
Inoffensive Centrist Democracy

Postby Altmoras » Fri Jan 07, 2022 11:42 pm

Ever-Wandering Souls wrote:
Refuge Isle wrote:95% of this thread has thus far been posting nation updating times. I have a high degree of confidence in those timestamps.

Variance is one thing, a two minute time span becoming a twelve minute timespan when all other points on update appear normal is worth investigation.


Certainly not saying it didn't happen, or that it shouldn't be investigated, more just that we're also performing a record number of actions per second for site history (with huge jumps and modern macros/tools), on two decade old spaghetti code, during an update that exists for an hour because 2 tin cans in Canada have to run at 100% capacity for that long to process it - so there's at least plausible other considerations, and we should be asking admin to investigate, before confidently going all in on that.


This is a BS narrative, if normal player actions were what was causing site instability we would have seen that instability while the 98 defenders in Artificial Solar System were opening 97 pages each to cross endorse, not after our 2 minute warning when we stopped crossing. We should have seen an improvement in site speeds at the 2 minute warning if anything. The lag also started well before we jumped, it was not caused by our jump, the lag started when our 2 minute warning stretched to 6 minutes not when we gave the go command.
Benevolent Thomas-Today at 11:15 AM
"I'm not sure if Altmoras has ever been wrong about anything."

User avatar
Vando0sa
Envoy
 
Posts: 321
Founded: Mar 08, 2014
Father Knows Best State

Postby Vando0sa » Sat Jan 08, 2022 4:08 am

Plasma Sage Ghestis wrote:Seems myself and potentially a few others had more specific issues pertaining to this topic, some activity feed and game behavior discrepancies. On the initial jump, I jumped with the nation I'm posting from here. At the time of the jump, this nation's activity feed contains my move into A Liberal Haven, a couple people endorsing me, an ejection and ban from the region by GonkDroid, a couple more people endorsing me, then my resignation and later subsequent re-admittance to the WA. In the interim there, I switched a couple times and tried jumping into the region again while the window to do so was still drug open by the massive site lag, then noticed that when I switched back onto this nation that it still behaves as though it's in A Liberal Haven and, from what I can tell, never actually moved to TRR. I confirmed that it can still endorse everyone endorsing Garchyland. My endorsements of these nations appear to still count towards their totals. Other people still appear to be able to endorse me in A Liberal Haven, and all of that activity pertaining to this nation in A Liberal Haven appears to be showing up in TRR's activity feed. It also appears as though this nation never updated in A Liberal Haven as a result of all that, so its influence hasn't changed and there's no corresponding message in the activity feed. The game still shows A Liberal Haven on the side bar as the region this nation is currently in, and I have a telegram from the website informing me of my ban and ejection from the region. Not really sure what to think of all that, but it's what it is. I can grab the relevant snips of activity feeds if they're helpful, otherwise I don't see the point of cluttering the thread when it's easy to click a nation and see the feed.



Oh hey! I just made another thread about this and another weird glitchy thing that happened.. we might have code gremlins..

We removed you from the banlist and banjected again so maybe it fixed it.. still weird though..
Kevät itkee talven töitä Käy hyinen tuulen henki Kevät itkee talven töitä Virta kantaa luita rantaan

User avatar
Budgie Snugglers
Lobbyist
 
Posts: 17
Founded: Jan 15, 2018
Corrupt Dictatorship

Postby Budgie Snugglers » Sat Jan 08, 2022 5:14 am

Using the order that nations updated in + the update happenings up to the 43rd minute of update, I made graphs - don't know if they are helpful. Data here.

Image
Image

User avatar
Eluvatar
Site Admin
 
Posts: 2518
Founded: Mar 31, 2006
New York Times Democracy

Postby Eluvatar » Sat Jan 08, 2022 8:57 am

Thank you for bringing this to our attention and for some helpful graphs.
To Serve and Protect: UDL

Eluvatar - Taijitu member

User avatar
Unibot III
Negotiator
 
Posts: 6558
Founded: Mar 11, 2011
Democratic Socialists

Postby Unibot III » Sat Jan 08, 2022 9:29 am

Those graphs are very illustrative. I can only remember a similar spike happening in back to back updates once in 2011, I forget the full context — Eluvatar would have the logs in #udl, searching for “Do not abort” (and “Hold the fucking line”).

Nothing else from my experience really matches this, and I think the possibility of manufactured lag or a DDOS attack should be taken seriously.

One way for the admins to know for sure is to move ALH to the back or front end of the update order. And see if the spikes continue, and when. :p I’m mostly joking.
Last edited by Unibot III on Sat Jan 08, 2022 9:29 am, edited 1 time in total.
[violet] wrote:I mean this in the best possible way,
but Unibot is not a typical NS player.
Milograd wrote:You're a caring, resolute lunatic
with the best of intentions.
Org. Join Date: 25/05/2008 | Former Delegate of TRR | Gameplay Alignment: -18 / -13
Unibotian Factbook // Collected works // The Gameplay Alignment Test //
Proudly Authored 9 GA Res., 14 SC Res. // Commended by SC#78

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
✯ Duty is Eternal, Justice is Imminent: UDL

User avatar
Plasma Sage Ghestis
Political Columnist
 
Posts: 2
Founded: Nov 28, 2021
Inoffensive Centrist Democracy

Postby Plasma Sage Ghestis » Sat Jan 08, 2022 9:34 am

Vando0sa wrote:
Plasma Sage Ghestis wrote:Seems myself and potentially a few others had more specific issues pertaining to this topic, some activity feed and game behavior discrepancies. On the initial jump, I jumped with the nation I'm posting from here. At the time of the jump, this nation's activity feed contains my move into A Liberal Haven, a couple people endorsing me, an ejection and ban from the region by GonkDroid, a couple more people endorsing me, then my resignation and later subsequent re-admittance to the WA. In the interim there, I switched a couple times and tried jumping into the region again while the window to do so was still drug open by the massive site lag, then noticed that when I switched back onto this nation that it still behaves as though it's in A Liberal Haven and, from what I can tell, never actually moved to TRR. I confirmed that it can still endorse everyone endorsing Garchyland. My endorsements of these nations appear to still count towards their totals. Other people still appear to be able to endorse me in A Liberal Haven, and all of that activity pertaining to this nation in A Liberal Haven appears to be showing up in TRR's activity feed. It also appears as though this nation never updated in A Liberal Haven as a result of all that, so its influence hasn't changed and there's no corresponding message in the activity feed. The game still shows A Liberal Haven on the side bar as the region this nation is currently in, and I have a telegram from the website informing me of my ban and ejection from the region. Not really sure what to think of all that, but it's what it is. I can grab the relevant snips of activity feeds if they're helpful, otherwise I don't see the point of cluttering the thread when it's easy to click a nation and see the feed.



Oh hey! I just made another thread about this and another weird glitchy thing that happened.. we might have code gremlins..

We removed you from the banlist and banjected again so maybe it fixed it.. still weird though..

Yep, looks like it worked on your second attempt there. I have the TG telling me I've been ejected and banned, I have TRR on my sidebar now, and I can no longer endorse nations in A Liberal Haven. Guess I'm definitely jumping again now :p

User avatar
[violet]
Site Admin
 
Posts: 15581
Founded: Antiquity

Postby [violet] » Sun Jan 09, 2022 3:14 pm

Eluvatar and I have both been looking into this. Thanks again also for the details and charts above, which make it much easier to figure out what we should be looking at.

System load has remained normal since Jan 5, when we blocked a Challenge bot. For the week prior, and especially on Dec 30, 31st, and Jan 3, load from this bot caused update to run long.

Since then, update has more or less hit target length, but it did experience an unusual stall on Jan 7. At this point, I suspect the problem lies in the Activity log, which became backlogged attempting to record a very large number of entries. We haven't identified the root cause, but so far the behavior hasn't repeated.

Update is the single most load-intensive point we have, so it is sensitive to other load-generating events that would otherwise go unnoticed.

User avatar
Ever-Wandering Souls
Negotiator
 
Posts: 6748
Founded: Jan 01, 2014
Father Knows Best State

Postby Ever-Wandering Souls » Sun Jan 09, 2022 4:37 pm

[violet] wrote:Eluvatar and I have both been looking into this. Thanks again also for the details and charts above, which make it much easier to figure out what we should be looking at.

System load has remained normal since Jan 5, when we blocked a Challenge bot. For the week prior, and especially on Dec 30, 31st, and Jan 3, load from this bot caused update to run long.

Since then, update has more or less hit target length, but it did experience an unusual stall on Jan 7. At this point, I suspect the problem lies in the Activity log, which became backlogged attempting to record a very large number of entries. We haven't identified the root cause, but so far the behavior hasn't repeated.

Update is the single most load-intensive point we have, so it is sensitive to other load-generating events that would otherwise go unnoticed.


Thank you for the update, the brief description of details, and for clarification that this was a technical issue with update hanging and not malicious activity.
Proud Raider; General of The Black Hawks, Ret.
TG me anytime; I'm always happy to talk about anything!

The Alicorns (Equestria) wrote:Let them stay, no need to badmouth them...From our view a bunch of nations just came in, seized the delegate position, and changed a few superficial things...we play NationStates differently...there's really no reason for us to be butthurt.
http://www.nationstates.net/page=rmb/postid=8944227
http://www.nationstates.net/page=rmb/postid=8951258

Misley wrote:
Hobbesistan wrote:Don't think I understand the question.
The color or what?..

Jesus, Hobbes, it's 2015. You can't just call someone "the color".

Reploid Productions wrote:Raiders are endlessly creative

How Do I Telegram API?

Omnis delenda est.

User avatar
[violet]
Site Admin
 
Posts: 15581
Founded: Antiquity

Postby [violet] » Sun Jan 09, 2022 5:54 pm

I've also contacted a few bot authors, because those have gotten out of control again. We have four in particular at the moment that are either illegal or broken (i.e. sending meaningless requests at maximum speed).

There's also something that identifies as"<Nation: NAME> Recruitment Tool." This looks legal and above board, but it's pretty inefficient, since it's being operated by multiple different nations, and they all look up the exact same Activity log data over and over every second or two. I don't know what this is or what it does, but it's a fair bit of load on that part of update.

User avatar
Roavin
Ambassador
 
Posts: 1172
Founded: Apr 07, 2016
Scandinavian Liberal Paradise

Postby Roavin » Sun Jan 09, 2022 6:06 pm

Thanks for the feedback, [v]! I do have some additional data to consider which might help you in your search:

Nakari's graphs only focus on the worst offender of the updates. However, of the four updates in which jumps into the region were performed, three were affected by this mysterious lag.

  • Jan 7 at 12am EST (major): A nominal 14 second trigger became 66 seconds long. Update rate slowed down from over 30 to about 3-9 nations per second at the point of the trigger.
  • Jan 7 at 12am EST (minor): This update did not show any irregularities.
  • Jan 8 at 12am EST (major): This update is the bad one shown in Nakari's graphs
  • Jan 8 at 12am EST (minor): A nominal 12 second trigger became 60 seconds long.

Collecting this data is a lot of work and it's very very difficult to do for minor updates (due to lack of a data dump), so here's additional data for that first major update. The start of the Rejected Realms updating was our estimated 2.5 minute warning for the target (by watching the activity feed until influence changes for TRR started). The nation Bladetress-22 was placed and estimated to update about 14 seconds before the target.

Relevant select happenings:
(1) 1/7/2022, 6:27:37 AM GMT+1: Bladetress-22's influence in The Rejected Realms rose from "Zero" to "Unproven".


Relevant data extracted from the region dump generated during that update:
  • TRR began updating at 06:24:51 GMT+1
  • Bernadina, the region immediately after TRR, began updating at 06:27:45 GMT+1
  • Bladetress-22 was at position 5687/6146 in TRR
  • 174 nations in 69 regions resided between TRR and A Liberal Haven
  • A Liberal Haven (which I'll abbreviate as ALH going forward) began updating at 06:28:41 GMT+1

TRR's start occurred is 2 minutes and 46 seconds away from the designated trigger. This is within normal range (we estimated 2.5 minutes). However, the time from the trigger to the first ALH happening was 66 seconds, much more than the estimated 14 seconds. Doing a bit of math:
  • 2 minutes and 46 seconds to update the first 5687 nations in TRR => 34.3 nations per second
  • 66 seconds for the 632 nations from trigger to ALH => 9.6 nations per second
  • 8 seconds for the 458 nations from trigger to end of TRR => 57.3 nations per second (actually faster!)
  • 58 seconds for the 174 nations between TRR and ALH => 3 nations per second

Had the rate remained at 34.3 nations per second, the trigger length would have been 18 seconds, which is within normal parameters. The only abnormality I can see is that there were 28 new single-nation regions between TRR and ALH, but that seems an unlikely culprit because then the constant overhead for starting a region update would have to be nearly a second, which is clearly not the case.

----

Also, to provide some context as to why some individuals think this may be foul play: This is a kind of spike that we haven't seen before, and it only happens right around the time a jump into an occupied region takes place. Even more suspiciously, of the four successive updates where a jump takes place, one of them is not affected, that being the weekday minor update which are traditionally the ones were the least amount of gameplayers are available (and, if this is a malicious actor, they would have to be a R/D gameplayer due to the timing). I hope it's just server gremlins, but I can't fault suspicions that it's not, and at this point we have neither proof that it is nor that it isn't, and should probably drop both speculations and undue celebrations until [v] and Elu know more.
Roavinur face from the South Pacific

"Not to mention TSP are some democracy wankers
and democracy wanking is peak defender
" — Tim

NS Coders Discord | I am a LOLcat | Former First Warden of TGW | aka Curious Observations

User avatar
Eluvatar
Site Admin
 
Posts: 2518
Founded: Mar 31, 2006
New York Times Democracy

Postby Eluvatar » Mon Jan 10, 2022 3:19 pm

[violet] wrote:At this point, I suspect the problem lies in the Activity log, which became backlogged attempting to record a very large number of entries. We haven't identified the root cause, but so far the behavior hasn't repeated.


I have some hypotheses for which data on happenings that happened more than 6 days ago would be helpful. As a gameplayer, I used to log all happenings for purposes of analyzing update, game mechanics, etc. If any players happen to have been doing that recently, I would much appreciate the submission of a log of happenings since the start of the year over GHR. If nobody has it, that's okay, there are other avenues for analysis they just might take longer.
To Serve and Protect: UDL

Eluvatar - Taijitu member

User avatar
[violet]
Site Admin
 
Posts: 15581
Founded: Antiquity

Postby [violet] » Mon Jan 10, 2022 4:45 pm

Roavin wrote:Also, to provide some context as to why some individuals think this may be foul play: This is a kind of spike that we haven't seen before, and it only happens right around the time a jump into an occupied region takes place. Even more suspiciously, of the four successive updates where a jump takes place, one of them is not affected, that being the weekday minor update which are traditionally the ones were the least amount of gameplayers are available (and, if this is a malicious actor, they would have to be a R/D gameplayer due to the timing). I hope it's just server gremlins, but I can't fault suspicions that it's not, and at this point we have neither proof that it is nor that it isn't, and should probably drop both speculations and undue celebrations until [v] and Elu know more.

There was a code change on December 2, 2021 that increased the workload caused by a region move. (The process now has to load the target region twice, not just once.) Doesn't line up particularly well with the issue here, but it's the only relevant system change I can think of. Potentially it might create noticeable slowdown when large numbers of nations are moving into the same region at the same time.

User avatar
Roavin
Ambassador
 
Posts: 1172
Founded: Apr 07, 2016
Scandinavian Liberal Paradise

Postby Roavin » Mon Jan 10, 2022 5:05 pm

[violet] wrote:There was a code change on December 2, 2021 that increased the workload caused by a region move. (The process now has to load the target region twice, not just once.) Doesn't line up particularly well with the issue here, but it's the only relevant system change I can think of. Potentially it might create noticeable slowdown when large numbers of nations are moving into the same region at the same time.


Yeah, it doesn't quite line up for two reasons that I can think of. First, the slowdown happened before all the nations started moving. Second, then we would have expected to see similar things happening in China on December 20, where the numbers were almost as large.
Roavinur face from the South Pacific

"Not to mention TSP are some democracy wankers
and democracy wanking is peak defender
" — Tim

NS Coders Discord | I am a LOLcat | Former First Warden of TGW | aka Curious Observations

Next

Advertisement

Remove ads

Return to Technical

Who is online

Users browsing this forum: No registered users

Advertisement

Remove ads