Delegate election not sticking

[Roavin]

Reporting it now because it's fresh, but this has happened a few times before (both to raiders and defenders)

In Nushlandia, a Delegate transition occurred due to R/D. This was marked on both national happenings on my nation Cupcake Roavin, as well as the world happenings feed:

12/30/2017, 6:57:04 AM GMT+1: Cupcake Roavin resigned from the World Assembly.
12/30/2017, 6:56:35 AM GMT+1: Cupcake Roavin became WA Delegate of Nushlandia.
12/30/2017, 6:56:35 AM GMT+1: Cupcake Roavin's influence in Nushlandia rose from "Zero" to "Power".
12/30/2017, 6:56:35 AM GMT+1: Cupcake Roavin relocated from Spear Danes to Nushlandia.

Since this was a same-second move, and other instances were also always very close, I could imagine that the nation got in as the region already began its update but hadn't started processing nations yet or something like that. Anyway, that's ostensibly a bug, so here's the report

[This name was taken]

That happens when 2 events happen, ostensibly, at the same time. Essentially, the server can't handle both, and one takes precedence over another. Like two Nations refounding a region at the same time (a la Hogwarts when LWU refounded it) or the missing Endo bug, where two Nations either enter the region simultaneously, so one doesn't register as 'in the region' or endorse the delegate/point similarly. From what I understand, this isn't really something that can be fixed, and is an inherent risk of R/D. It's a long standing issue/tradition.

[Vulturret]

That happens when 2 events happen, ostensibly, at the same time. Essentially, the server can't handle both, and one takes precedence over another. Like two Nations refounding a region at the same time (a la Hogwarts when LWU refounded it) or the missing Endo bug, where two Nations either enter the region simultaneously, so one doesn't register as 'in the region' or endorse the delegate/point similarly. From what I understand, this isn't really something that can be fixed, and is an inherent risk of R/D. It's a long standing issue/tradition.

I don't think it can be fixed, if the server can't handle two actions simultaneously, one just must occur prior to the other action. Even if it were to be fixed, there's massive problems with doing that, Roavin. In theory, they'd have to establish a priority list for actions that occur simultaneously. If such a priority list were to be established, no one would ever be totally content with the order, because one can always come up with some reason why it "benefits raiders" or is "unfair to defenders," or something of that regard. If some sort of priority list were to be established so the server just doesn't pick, it would just be a pain in the ass for the admins to deal with all the complaints. The nice thing about the server in the current form is that this bug/feature (dependent upon how you look at it) equally impacts either side of the raid/defense aisle, a random server selection that must be made during update at least can't be used to claim server bias against certain types of gameplayers.

[The Gipper]

I don't think the "bug" is that when it receives two inputs at basically the same time, the server has to decide which went first.

As I am reading the OP, the "bug" is that the game first announced that the nation became delegate, when it really didn't. So whatever part of the code that triggers "NationX became WA Delegate of RegionY" is behaving buggy, because the message was triggered in a situation when NationX actually didn't become delegate.

I seem to recall during the Milo coup in TSP that Elu said offsite he thought a nation had to be in a region when the region first started updating, and still had to be there when the (hopefully) new Delegate nation was processed - as that is when the nation's endorsements are counted. That would make what Roavin is saying make sense about update having already began when the move occurred but all nations weren't yet done processing.

Still, I'm with Roavin that if the game outputs a message that a nation became delegate when the nation did not actually become delegate - that sounds like an avoidable bug.

[[violet]]

That kind of thing is certainly a bug and can be addressed. It happens when there's a gap in between an event checking whether it should proceed and then actually completing: in between, another request can slip in, and it overwrites most of the effect of the first one.

You need to report what's happening in a lot of detail, though, for me to figure it out. I'm not sure what you're saying did or should have actually happened in the Cupcake Roavin situation. Who moved where when and what was the expected outcome?

[Roavin]

So, I don't know how the internals work (obviously). For the analysis, I'll be making some assumptions, based on what I know and seems most logical, though these assumptions may be false (I'll rejig if necessary). I will assume the following:

• The order of events entered into the world happenings feed represents the normative order in which the main game engine processed things.
• The update process runs asynchronously, and enters events into the world happenings feed as it encounters them.
• The regional update will take a snapshot of the nations in a region, and then updates those nations, thereby ignoring any and all nations moving in or out in the meanwhile.
• The actual start of the regional update is unknowable, but is atomic with respect to the game engine, and so will have been between some two consecutive events and not "on top of" an event (if this assumption isn't true, that might just be the bug - see below)
• Ties in Delegate elections are decided thusly: If there is an incumbent, the incumbent wins. Otherwise, the first nation to update wins (so the code in the update loop is something like "

  if (curr_nation.endocount > 0 && (!next_delegate || curr_nation.endocount > next_delegate.endocount)) { next_delegate = nation; }

  ").

Here are the happenings from Nushlandia, starting with the first move and ending with the last move (which happened after the Delegate election), reversed and numbered:

1: 12/30/2017, 6:56:33 AM GMT+1: Liliarchy ancillary 64 relocated from Plum Island to Nushlandia.
2: 12/30/2017, 6:56:34 AM GMT+1: Like a good neighbor Vinny was there and relocated from Spear Danes to Nushlandia.
3: 12/30/2017, 6:56:35 AM GMT+1: Cupcake Roavin relocated from Spear Danes to Nushlandia.
4: 12/30/2017, 6:56:35 AM GMT+1: 57th Overlanders Brigade relocated from Spear Danes to Nushlandia.
5: 12/30/2017, 6:56:35 AM GMT+1: Liliarchy ancillary 64's influence in Nushlandia rose from "Zero" to "Negotiator".
6: 12/30/2017, 6:56:35 AM GMT+1: Like a good neighbor Vinny was there and's influence in Nushlandia rose from "Zero" to "Eminence Grise".
7: 12/30/2017, 6:56:35 AM GMT+1: 57th Overlanders Brigade's influence in Nushlandia rose from "Zero" to "Eminence Grise".
8: 12/30/2017, 6:56:35 AM GMT+1: Cupcake Roavin's influence in Nushlandia rose from "Zero" to "Power".
9: 12/30/2017, 6:56:35 AM GMT+1: Cupcake Roavin became WA Delegate of Nushlandia.
10: 12/30/2017, 6:56:35 AM GMT+1: Biscuit-Raider 23 relocated from Plum Island to Nushlandia.
11: 12/30/2017, 6:56:39 AM GMT+1: Totally Not Merlin 30 relocated from Spear Danes to Nushlandia.

First, you'll note that most of the things happened in the same second, so a race condition is reasonable to assume here.

Involved were the following nations and their endorsements (I'm skipping endorsements from other nations that did not move into Nushlandia at all):

• Liliarchy ancillary 64: Biscuit-Raider 23 (1 endorsement)
• Like a good neighbor Vinny was there and: Totally Not Merlin 30, Cupcake Roavin (2 endorsements)
• Cupcake Roavin: Totally Not Merlin 30, Like a good neighbor Vinny was there and, 57th Overlanders Brigade (3 endorsements)
• 57th Overlanders Brigade: Like a good neighbor Vinny was there and (1 endorsement)
• Biscuit-Raider 23: (none)
• Totally Not Merlin 30: 57th Overlanders Brigade, Cupcake Roavin, Like a good neighbor Vinny was there and (3 endorsements)

Before the first happening, the region had no WA nations. Now, the happenings one by one, plus at the completion of that event, the relevant nations in region, their effective endorsements, and who would have been candidates for the Delegacy election:

1: 12/30/2017, 6:56:33 AM GMT+1: Liliarchy ancillary 64 relocated from Plum Island to Nushlandia.

Nations and effective endorsements:

• Liliarchy ancillary 64: (none)

Delegate candidates: (none)

2: 12/30/2017, 6:56:34 AM GMT+1: Like a good neighbor Vinny was there and relocated from Spear Danes to Nushlandia.

Nations and effective endorsements:

• Liliarchy ancillary 64: (none)
• Like a good neighbor Vinny was there and: (none)

Delegate candidates: (none)

3: 12/30/2017, 6:56:35 AM GMT+1: Cupcake Roavin relocated from Spear Danes to Nushlandia.

Nations and effective endorsements:

• Liliarchy ancillary 64: (none)
• Like a good neighbor Vinny was there and: Cupcake Roavin
• Cupcake Roavin: Like a good neighbor Vinny was there and

Delegate candidates: Like a good neighbor Vinny was there and, Cupcake Roavin

4: 12/30/2017, 6:56:35 AM GMT+1: 57th Overlanders Brigade relocated from Spear Danes to Nushlandia.
Nations and effective endorsements:

• Liliarchy ancillary 64: (none)
• Like a good neighbor Vinny was there and: Cupcake Roavin
• Cupcake Roavin: Like a good neighbor Vinny was there and, 57th Overlanders Brigade
• 57th Overlanders Brigade: Like a good neighbor Vinny was there and

Delegate candidates: Cupcake Roavin

5: 12/30/2017, 6:56:35 AM GMT+1: Liliarchy ancillary 64's influence in Nushlandia rose from "Zero" to "Negotiator".
6: 12/30/2017, 6:56:35 AM GMT+1: Like a good neighbor Vinny was there and's influence in Nushlandia rose from "Zero" to "Eminence Grise".
7: 12/30/2017, 6:56:35 AM GMT+1: 57th Overlanders Brigade's influence in Nushlandia rose from "Zero" to "Eminence Grise".
8: 12/30/2017, 6:56:35 AM GMT+1: Cupcake Roavin's influence in Nushlandia rose from "Zero" to "Power".

The region is updating, and since the influence changed for all four nations up to that point, they were all included in the snapshot made when the region started updating. So therefore, the start of the regional update should have been between events 4 and 5.

9: 12/30/2017, 6:56:35 AM GMT+1: Cupcake Roavin became WA Delegate of Nushlandia.

The Delegate election concluded, electing Cupcake Roavin (as it should, since it was the sole nation with the most effective endorsements in region).

10: 12/30/2017, 6:56:35 AM GMT+1: Biscuit-Raider 23 relocated from Plum Island to Nushlandia.
Nations and effective endorsements:

• Liliarchy ancillary 64: Biscuit-Raider 23
• Like a good neighbor Vinny was there and: Cupcake Roavin
• Cupcake Roavin: Like a good neighbor Vinny was there and, 57th Overlanders Brigade
• 57th Overlanders Brigade: Like a good neighbor Vinny was there and
• Biscuit-Raider 23: (none)

Delegate candidates: Cupcake Roavin

This was a straddler arriving after the Delegate election; I'm listing it separately because it came in the second. But normatively, it shouldn't have affected the election because not only did it come too late, it also only bumped the endorsement count of Liliarchy ancillary 64 to 1, while Cupcake Roavin already had two endorsements.

Expected behavior after Event 9:

• Cupcake Roavin is elected Delegate via happening
• Cupcake Roavin is shown as Delegate on the regional page
• Cupcake Roavin is entered as Delegate on the regional history page
• Cupcake Roavin has all regional controls.

Actual behavior (observed after event 10):

• Cupcake Roavin is elected Delegate via happening
• Cupcake Roavin is not shown as Delegate on the regional page
• Cupcake Roavin is not entered as Delegate on the regional history page
• Cupcake Roavin does not have regional controls.

Now, I don't know enough internals to make a good guess, obviously . One thing stands out, though: Cupcake Roavin was the nation at the latest point in the update order, and it got the endorsement to bump it over to make it the clear Delegate candidate right at the end. That endorsement was by 57th Overlanders Brigade, which also updated after the other two. If the begin of the regional update is not atomic with respect to the game engine (for example, it makes a snapshot twice in quick succession and does different processes with different snapshots), then this could be explained by the region update being initiated concurrent with event 4 - one snapshot would elect the Vinny nation, as that one tied with Cupcake Roavin but updates earlier, while the other snapshot sees the missing endorsements and clearly elects Roavin. As the snapshots "come together" somehow, a mismatch is detected and therefore the game simply doesn't recognize a Delegate, even though one of them already announced its election via happening.

Whew.

[Caelapes]

For future reference, the start of a region’s update isn’t unknowable. That’s what the LASTUPDATE API shard shows, or the time stamp from the “Last WA Update” on a region’s page. It’s not possible to go back and get that data later... although that would be nice.

It doesn’t give a Happening, so we can’t compare happening IDs to know if the region had begun to update before nations moved in, though.

[Roavin]

Right - I meant in relation to the order of events in world happenings. I should have been more explicit, sorry

[Unibot III]

I'm not sure that's the standard "race" bug. To have gained influence in the region and have its endorsements register, Cupcake Roavin had to have updated. Nations don't update with the race bug. In that sense, what has transpired is weirder than that. Cupcake Roavin updated without having registered as having ever moved into the region?

[[violet]]

Here are the happenings from Nushlandia, starting with the first move and ending with the last move (which happened after the Delegate election), reversed and numbered:

Thanks, that's very clear. I'm assuming there was no Delegate displayed at the end of all this.

What's almost certainly happened is Biscuit-Raider 23's arrival occurred at the same instant the region was updating, like this:

1. Daily update loads region
2. Biscuit-Raider 23 movement routine loads region
3. Daily update saves region with updated Delegate info
4. Biscuit-Raider 23 movement routine saves region

Which is bad because #4 overwrites #3, and doesn't contain the updated Delegate election info.

[Roavin]

Oooh. Yeah, if they're not locked against each other, then that does seem like the most likely issue (and much simpler than what I suggested even). And yeah, no Delegate was displayed, and it wasn't entered into the history.

So, uh ... can I suggest flock()?

[Eluvatar]

Yes, flock would be helpful (although it'd come with a small performance penalty).

I should note that the window for this sort of shenanigan is not as wide as the region's entire update (otherwise the feeders would never change delegate).

[[violet]]

We use flock to avoid contention on a single resource. This situation is a little more complicated because it involves the interplay of a couple of different systems: in #2 above, the movement routine is fast-loading the region out of an in-memory cache while #3 is writing to disk. But there is something in this situation holding open the window for a race condition much longer than it needs to, so I should be able to address that.

[Roavin]

"There are only two hard things in Computer Science: cache invalidation and naming things."

-- Phil Karlton

A transactional approach would do the trick to get rid of 'em completely. But that might be hard to retrofit if some global state is updated simultaneously etc.