[ADMIN MESSAGE] Ongoing Database Problems

[United Calanworie]

Looking for a place to discuss the ongoing outage? Use this thread. The one you're currently on is a locked static announcement feed.

---

FAQs:

• What data was lost? Read the announcements below.
• It says access denied. Am I IP banned? No.
• When was the last backup? 11 May 2023
• What happened? We're still unclear as to the root cause, but the primary nVME drive backing the database server suddenly failed, and then was unable to be recovered.
• I was wondering if {data not in the list above} was lost...? No.
• Has the CTE period been extended due to this drive failure? Yes. The CTE timer will be at 35 days for the next while. (Vacation mode timers will be at 67 days.)
• You said polls were lost. When were they recoverable from? 11 May 2023, per our backup.

Update: 29 Apr 2024
Daily update is running as normal now. Recovery on the failed disk is progressing in a much more positive manner than expected. We currently expect (but cannot guarantee) to be able to recover the lost Cards data, as well as potentially much of the other lost data.

Update: 25 Apr 2024
The next daily update may occur at 2024-04-26, 11:00:00 p.m. PDT, though it may be delayed (it will not be earlier). The news page has been updated to reflect this and translate it to all timezones.

Update: 21 Apr 2024

Since the last update we've succeeded in restoring several other tables with some of us (literally) working through the night to do so. Progress continues on some larger ones that are reluctant to budge and actually let us load them in. There's three large tables (I'm talking tens to hundreds of millions of rows) remaining that haven't yet been loaded back in, but they're on our priority list to get sorted. Our current plan is to bring up the game once we have everything restored (minus cards) and then restore cards once we have a better idea of whether or not we're going to be able to recover data associated with it.

Update: 20 Apr 2024

With the exception of a few tables, we've gotten *most* of the data imported. We're currently working on restoring the lost data to the best of our ability, rebuilding indexes, etc. We still don't have an actual ETA as to when the site will be back up, but things are looking slightly better than they were a few days ago.

Update: 19 Apr 2024

Hi all, the forums are flickering in and out of maintenance mode -- we've got approximately 1200 people attempting to load them at any given point in time right now and it turns out that's too many people. The more you know.

With that said, I'm going to push an announcement re. the database recovery process:
- We've got MySQL back up and running
- We're importing data
- No, we don't have a timeline when the site will be back up
- Yes, we will keep you posted

Update on April 18th forum blackout

The forum automatically entered maintenance mode early this morning (UTC) as a result of higher-than-expected load. We've tweaked a few settings and it's all back now. No posts, signatures, or any other forum data was lost.

---

On 14 Apr 2024, we experienced a database failure. This is the most recent update from our team as the situation progresses.

Good morning everyone. This is a bit of a longer announcement, and it quickly describes where we're at, what kind of data loss we think we'll have, and steps that we're trying to take to mitigate it.

The disk that failed contained several tables on it, but the ones of primary concern to most players are going to be:

• Telegram API keys
• Telegram templates (not the text of the templates themselves, this is an accessory table used to help speed along processes involving templated telegrams being sent)
• The telegram queue (the status of recruitment/campaign telegrams that had recently been sent out)
• Recruitment telegram stats
• Polls and associated data
• And the biggest one: Cards.

I'm going to talk about cards first here as that's the largest community that's going to be affected the most. We've likely lost months worth of data in terms of:

• card ownership
• trades
• bank data
• collections
• bids, asks, auctions

There's a potential that we will be able to recover some of that data. Unfortunately, the disk failed in a rather spectacular manner, and is resistant to our attempts to recover anything from it. We will keep you updated.

As to the telegrams information:
The API keys are gone. You'll have to file a GHR to get a new key once we're back online. Hopefully that's not too much of an issue. We're looking into rebuilding the other tables from the main telegrams table, as it contains all the information that is required to build the other tables. It should be possible to recover relatively quickly. It's likely that mass telegrams will be restored to whatever state they were in up to approximately a week ago for the largest telegrams.

We're continuing our efforts at data recovery, and will let you know when we know more re. data loss. In the meantime, we're also trying to bring the game back up with critically impacted areas (e.g. cards, telegrams) disabled while we do our best to restore the data that we can.

[Katganistan]

As we look forward to resuming normal operations on NationStates, it’s important to understand the process for obtaining a new client key via Getting Help Request for sending telegrams via API. If you're recruiting for a large established region, check with your Regional Leaders first: they probably have a key you can request to use.

If you wish to use a key for Campaigning, it's best to Found a region with a puppet and use that puppet to request a key. Recruitment keys are specific to regions, but you can use a key with any of your puppets for campaigning anywhere.


Please follow these instructions carefully:

Order of Information:
Provide the necessary details in the following order:

Your Region Name:
Your Nation Name:
Purpose of the Key: (e.g., recruitment, WA campaigns, etc.)

Accuracy Matters:
Ensure that you spell your nation and region names correctly. Accuracy is crucial for processing your request.
If there is a "The" in front of your region name, include it in the request.
Please do NOT include the pre-title when giving us your nation name. (that is, YourNation, not "The Title of Awesomeness of YourNation"


Eligibility:
To receive the key, you must be in the region for which you are requesting it.

If you are looking for information on how to send telegrams via API, Ever-Wandering Souls has created a thread that should help with that. Remember that you MUST check the Recruitment box on your telegram template if recruiting, or the WA Campaign box if campaigning.

And a note about Client Keys:

I just cleaned up the Client Key database. I removed

• Anyone whose script hasn't been active for a year or more
• Anyone who had requested a key three or more months ago and never used it

We don't mind issuing Client Keys to players who will use them, but it's annoying and a fair amount of work to issue keys to people who just think it's cool to have a key.

Please don't request a Client Key unless you a) know how to use it and b) plan to use it.

Thanks,
The NS Game Mods