Delegacy Bug

[Nreremeopahpera]

My nation, nreremeopahpera participated in a raider/defender gameplay liberation this past minor update of the region The New Order Last days of Europe.

Sure enough, my nation ended up with more endorsements than the raider delegate and became the delegate, but only temporarily. The delegate immediately reverted back to being under control by the raiders and I have no access to the regional controls.

Regional happenings showing my nation taking the delegacy:


National happenings of my nation taking the delegacy:


Raider national happenings:


It seems to me that this is a bug and my nation should still be the delegate of the region. Wondering if I missed anything. Thanks!

[Miravana]

Heya! I’d like to point out that this glitch has happened many times on tag runs, I don’t know if that has been reported before because we generally don’t care about the smaller stakes. With it happening in a larger scale operation, I hope that we can focus on fixing the glitch in general if possible instead of fixing the delegacy and calling it a day

Edit: Apparently it has been reported before, just no response to that thread
viewtopic.php?f=15&t=486780#p37241971

[Roavin]

Here's another thread about the topic from 2018, with a more detailed analysis from me as well as participation from [v] and Elu. [v] gives some insight into how that happens, and it mostly seems to fit the situation here. Basically, it's ostensibly that good ol' race condition again, one of the hardest flavors of bugs to fix, so I understand why it's still there, bugging gameplayers on all sides occasionally.

There's a difference here, though: Many people in Libcord did see OP's nation as the Delegate, so it seems that the game saw that nation as Delegate for several seconds before being 'overwritten' with the previous stale value. It's hard to say how long that was, but it must certainly have been at least a few seconds. Based on what [v] wrote here, that would imply that there was one operation that operated asynchronously from update and took a considerable amount of time. The delegacy change was happening ID 16826090 @ 2021-11-28 17:03:33+0000. A banjection occured with happening 16826133 @ 2021-11-28 17:03:41+0000, 8 seconds later, with the previous Delegate, so it's reasonable to assume that the action that caused the stale region write occurred between those two happenings:

11/28/2021, 6:03:42 PM GMT+1: European Bison Empire relocated from The New Order Last days of Europe to The Rejected Realms.
11/28/2021, 6:03:41 PM GMT+1: European Bison Empire was ejected and banned from The New Order Last days of Europe by Tamiel.
11/28/2021, 6:03:40 PM GMT+1: Christian Lindner II relocated from Artificial Solar System to The New Order Last days of Europe.
11/28/2021, 6:03:39 PM GMT+1: Emu Tribe endorsed Frinil.
11/28/2021, 6:03:37 PM GMT+1: Controlitia WA relocated from Artificial Solar System to The New Order Last days of Europe.
11/28/2021, 6:03:37 PM GMT+1: Preskovia 6 relocated from Artificial Solar System to The New Order Last days of Europe.
11/28/2021, 6:03:37 PM GMT+1: Reballic endorsed Frinil.
11/28/2021, 6:03:36 PM GMT+1: Stalenskaya Puppeteer endorsed Frinil.
11/28/2021, 6:03:34 PM GMT+1: Eolusia1 endorsed Frinil.
11/28/2021, 6:03:33 PM GMT+1: Astrophysics endorsed Lebazgh.
11/28/2021, 6:03:33 PM GMT+1: Grape Island endorsed Frinil.
11/28/2021, 6:03:33 PM GMT+1: Nreremeopahpera seized the position of The New Order Last days of Europe WA Delegate from Tamiel.

Those are just endorsements and moves, though, which we had always assumed to be "quick" unless one of those took longer than it otherwise would. Another possibility, I guess, would be that the banjection is the culprit, since that process logically loads two regions (this one plus TRR), and looking at the happenings both before and after the region updated, the invader delegate was banjecting incoming nations at about 1 nation every 2-4 seconds rather than the 14 seconds it took for between the last pre-update banject and the first post-update banject. So if there was a lag spike with a banject there that caused the stale region write, that would explain what we saw.

(theoretically, the culprit could be a happening issued before the delegacy change and written to storage after it, but I considered that scenario unlikely given how long it took for the stale region write)

EDIT: Tamiel just confirmed to me on Discord that they didn't have lag spikes and got several "unauthorized" error messages. That, plus the detail I missed about the 26 verified endorsements, might point to something else than the banjection, and rather something like ... an exception during update?

(see i told you liberations are literally impossible )

[[violet]]

Thanks for the report. Just acknowledging that we're looking at this. It's a hard one to pin down, as per the above analysis, which is exactly right.

[[violet]]

EDIT: Tamiel just confirmed to me on Discord that they didn't have lag spikes and got several "unauthorized" error messages. That, plus the detail I missed about the 26 verified endorsements, might point to something else than the banjection, and rather something like ... an exception during update?

This is a shame, because banjection does look like a reasonably likely culprit, triggering multiple writes.

Anyway, how reproducible do we think this is? Because I can set up a test region that writes a log every time it saves to say why it's doing it. That's not something I can impose on every region in the game without slowing down update, but if it's just this one region, and we manage to trigger the bug, I should be able to figure out what happened.

[Tim-Opolis]

EDIT: Tamiel just confirmed to me on Discord that they didn't have lag spikes and got several "unauthorized" error messages. That, plus the detail I missed about the 26 verified endorsements, might point to something else than the banjection, and rather something like ... an exception during update?

This is a shame, because banjection does look like a reasonably likely culprit, triggering multiple writes.

Anyway, how reproducible do we think this is? Because I can set up a test region that writes a log every time it saves to say why it's doing it. That's not something I can impose on every region in the game without slowing down update, but if it's just this one region, and we manage to trigger the bug, I should be able to figure out what happened.

We reproduce it multiple times a night any time we’re chasing tag raiders, would you like us to document and record it whenever it happens? I can comfortably say there isn’t a single night of tag chasing that doesn’t have at least one delegacy glitch. We can do out best to reproduce it in a test region, but it’d require raider cooperation for live fire conditions and you’d need a lot of repetition to consistently catch the glitch.

[[violet]]

We reproduce it multiple times a night any time we’re chasing tag raiders, would you like us to document and record it whenever it happens?

A few more examples would be handy, thanks (region names + times). You don't need to do them all. Unless some high percentage of the examples include banjection; if that's the case, I'd like to see how common it is.

I can comfortably say there isn’t a single night of tag chasing that doesn’t have at least one delegacy glitch. We can do out best to reproduce it in a test region, but it’d require raider cooperation for live fire conditions and you’d need a lot of repetition to consistently catch the glitch.

We may need that, because thorough logging is the best way to catch what's happening, but it's too slow to enable for all regions during update.

[Altmoras]

This is a shame, because banjection does look like a reasonably likely culprit, triggering multiple writes.

Anyway, how reproducible do we think this is? Because I can set up a test region that writes a log every time it saves to say why it's doing it. That's not something I can impose on every region in the game without slowing down update, but if it's just this one region, and we manage to trigger the bug, I should be able to figure out what happened.

We reproduce it multiple times a night any time we’re chasing tag raiders, would you like us to document and record it whenever it happens? I can comfortably say there isn’t a single night of tag chasing that doesn’t have at least one delegacy glitch. We can do out best to reproduce it in a test region, but it’d require raider cooperation for live fire conditions and you’d need a lot of repetition to consistently catch the glitch.

It should be noted that this particular permutation of the bug is slightly different to the typical one, the one I typically see has a happening generated on my reports page about someone becoming delegate in a tag raid target, but nobody actually becomes delegate, and the region stays at "12 hours ago" even though nations that moved into it on time will have influence changes.

[Refuge Isle]

We reproduce it multiple times a night any time we’re chasing tag raiders, would you like us to document and record it whenever it happens?

A few more examples would be handy, thanks (region names + times). You don't need to do them all. Unless some high percentage of the examples include banjection; if that's the case, I'd like to see how common it is.

Yeah, banjection isn't a component of the tag raids where we usually see this. A more shared attribute is some 15 nations moving into a region within 1 second of it updating. The New Order Last Days of Europe was unique in the regard that there was a lot more time to witness the issue, as though it were happening in slow motion.

I can comfortably say there isn’t a single night of tag chasing that doesn’t have at least one delegacy glitch. We can do out best to reproduce it in a test region, but it’d require raider cooperation for live fire conditions and you’d need a lot of repetition to consistently catch the glitch.

We may need that, because thorough logging is the best way to catch what's happening, but it's too slow to enable for all regions during update.

Alternatively, enable it for all regions. But somehow in a situation where there are only some 30 regions that update substantially faster than normal -- raid constantly until you get it

[Roavin]

It should be noted that this particular permutation of the bug is slightly different to the typical one, the one I typically see has a happening generated on my reports page about someone becoming delegate in a tag raid target, but nobody actually becomes delegate, and the region stays at "12 hours ago" even though nations that moved into it on time will have influence changes.

That's true, though I've seen this flavor as well (and I believe that's the one I reported back in 2018). What seemed different to me this time was rather how long the ostensibly "correct" delegate showed before it reverted back to the old one, though this may just be a function of region size (tag targets are usually puny).

We may need that, because thorough logging is the best way to catch what's happening, but it's too slow to enable for all regions during update.

Would it help if, say, a specific update is agreed upon beforehand for a tag run + chase, with the raiders sending you their target list beforehand? If so, how early beforehand would you have to be informed of the target list?

[Miravana]

It should be noted that this particular permutation of the bug is slightly different to the typical one, the one I typically see has a happening generated on my reports page about someone becoming delegate in a tag raid target, but nobody actually becomes delegate, and the region stays at "12 hours ago" even though nations that moved into it on time will have influence changes.

Actually, this particular permutation is exactly like the on we get often times when we are tagging. The reason nobody becomes delegate on your side is because there was never a delegate to begin with. I'd assume when you are chasing you don't see the region exactly when it updates because you are busy chasing. Often a trigger observing a target who isn't jumping will see the point become delegate and then it return to nobody being the delegate. I will try to have the regions collected if we observe this in future tags.

Would it help if, say, a specific update is agreed upon beforehand for a tag run + chase, with the raiders sending you their target list beforehand? If so, how early beforehand would you have to be informed of the target list?

This could definitely be arranged if you'd be wiling to spectate an update

[[violet]]

This could definitely be arranged if you'd be wiling to spectate an update

Okay this sounds like a plan.

Give me a little time write some logging code, then we can arrange an update or two where somebody emails me a list of target regions a few minutes beforehand. Hopefully one will exhibit the bug, and then I can see what's happening.

[[violet]]

I've also added a tweak that might help reduce the frequency of the problem... let me know if you notice any change.

[Miravana]

This could definitely be arranged if you'd be wiling to spectate an update

Okay this sounds like a plan.

Give me a little time write some logging code, then we can arrange an update or two where somebody emails me a list of target regions a few minutes beforehand. Hopefully one will exhibit the bug, and then I can see what's happening.

Alright sounds good, we can probably manage to target up to 100 regions per update depending on if more = better chances of getting the bug to appear, I'd also assume the more individual people per region the better, as the bug might be triggered easier with more people participating in a given raid. And yeah I'd be able to email the targets before the raid, just let me know where to send em beforehand

[Miravana]

I've also added a tweak that might help reduce the frequency of the problem... let me know if you notice any change.

For the record, we didn't see any delegacy bugs today at update, and we had a lot of updaters and targets on both sides, so its possible this tweak did help, we'll see how things continue to go moving forward

[[violet]]

For the record, we didn't see any delegacy bugs today at update

That's exciting! Thanks, keep me posted.