FAQs:
- What data was lost? Read the announcements below.
- It says access denied. Am I IP banned? No.
- When was the last backup? 11 May 2023
- What happened? We're still unclear as to the root cause, but the primary nVME drive backing the database server suddenly failed, and then was unable to be recovered.
- I was wondering if {data not in the list above} was lost...? No.
- Has the CTE period been extended due to this drive failure? Yes. The CTE timer will be at 35 days for the next while. (Vacation mode timers will be at 67 days.)
- You said polls were lost. When were they recoverable from? 11 May 2023, per our backup.
Update: 29 Apr 2024
Daily update is running as normal now. Recovery on the failed disk is progressing in a much more positive manner than expected. We currently expect (but cannot guarantee) to be able to recover the lost Cards data, as well as potentially much of the other lost data.
Update: 25 Apr 2024
The next daily update may occur at 2024-04-26, 11:00:00 p.m. PDT, though it may be delayed (it will not be earlier). The news page has been updated to reflect this and translate it to all timezones.
Update: 21 Apr 2024
United Calanworie wrote:Since the last update we've succeeded in restoring several other tables with some of us (literally) working through the night to do so. Progress continues on some larger ones that are reluctant to budge and actually let us load them in. There's three large tables (I'm talking tens to hundreds of millions of rows) remaining that haven't yet been loaded back in, but they're on our priority list to get sorted. Our current plan is to bring up the game once we have everything restored (minus cards) and then restore cards once we have a better idea of whether or not we're going to be able to recover data associated with it.
Update: 20 Apr 2024
United Calanworie wrote:With the exception of a few tables, we've gotten *most* of the data imported. We're currently working on restoring the lost data to the best of our ability, rebuilding indexes, etc. We still don't have an actual ETA as to when the site will be back up, but things are looking slightly better than they were a few days ago.
Update: 19 Apr 2024
United Calanworie wrote:Hi all, the forums are flickering in and out of maintenance mode -- we've got approximately 1200 people attempting to load them at any given point in time right now and it turns out that's too many people. The more you know.
With that said, I'm going to push an announcement re. the database recovery process:
- We've got MySQL back up and running
- We're importing data
- No, we don't have a timeline when the site will be back up
- Yes, we will keep you posted
Update on April 18th forum blackout
SherpDaWerp wrote:The forum automatically entered maintenance mode early this morning (UTC) as a result of higher-than-expected load. We've tweaked a few settings and it's all back now. No posts, signatures, or any other forum data was lost.
On 14 Apr 2024, we experienced a database failure. This is the most recent update from our team as the situation progresses.
United Calanworie wrote:Good morning everyone. This is a bit of a longer announcement, and it quickly describes where we're at, what kind of data loss we think we'll have, and steps that we're trying to take to mitigate it.
The disk that failed contained several tables on it, but the ones of primary concern to most players are going to be:
- Telegram API keys
- Telegram templates (not the text of the templates themselves, this is an accessory table used to help speed along processes involving templated telegrams being sent)
- The telegram queue (the status of recruitment/campaign telegrams that had recently been sent out)
- Recruitment telegram stats
- Polls and associated data
- And the biggest one: Cards.
I'm going to talk about cards first here as that's the largest community that's going to be affected the most. We've likely lost months worth of data in terms of:There's a potential that we will be able to recover some of that data. Unfortunately, the disk failed in a rather spectacular manner, and is resistant to our attempts to recover anything from it. We will keep you updated.
- card ownership
- trades
- bank data
- collections
- bids, asks, auctions
As to the telegrams information:
The API keys are gone. You'll have to file a GHR to get a new key once we're back online. Hopefully that's not too much of an issue. We're looking into rebuilding the other tables from the main telegrams table, as it contains all the information that is required to build the other tables. It should be possible to recover relatively quickly. It's likely that mass telegrams will be restored to whatever state they were in up to approximately a week ago for the largest telegrams.
We're continuing our efforts at data recovery, and will let you know when we know more re. data loss. In the meantime, we're also trying to bring the game back up with critically impacted areas (e.g. cards, telegrams) disabled while we do our best to restore the data that we can.