Suggestion - Download RMB Archive

[Misley]

Split from here at Fris's suggestion:

I see a lot of players saying that one reason they don't want to refound is because they'd lose the RMB of the region. So ... general question for consideration:
Would having a way to download your region's RMB make the possibility of transition more palatable?

Absolutely. One of the biggest concerns we had with refounding The Internationale was saving its RMB. I managed to do so by downloading the Forum View and Region API message shards which combine to give a timestamp, nation name, raw message, and post ID. If all of that information could be packaged by the game and downloaded in one go, it'd be great.

It'd be even better if there was a way to "import" past RMBs, because as it currently stands all old posts just vanish from the server. If we could restore those old posts upon refound so they could be viewed directly on NS (and not just on another website), that would be really cool.

[Riftend]

I do like it myself - plenty of regions I haven't gone and refounded because i'd never see the RMB again~

[Mousebumples]

Bearing in mind that there is no way that I have the tech knowledge to say what is or is not feasible, what sort of access should this sort of thing have?

Should it be restricted to the Founder (and exec-WAD where applicable), where regions that are interested in this sort of thing can semi-occasionally download the RMB Archive for their records? ( similar to backing up your computer every now and again in case of malfunction)

Should any nation that fits the definition of "native" (more influence in RegionA than any other region) be able to download it, even if they're not in the region anymore?

Other than "this would be nice" ... anything else worth focusing in on?

[Riftend]

Bearing in mind that there is no way that I have the tech knowledge to say what is or is not feasible, what sort of access should this sort of thing have?

Should it be restricted to the Founder (and exec-WAD where applicable), where regions that are interested in this sort of thing can semi-occasionally download the RMB Archive for their records? ( similar to backing up your computer every now and again in case of malfunction)

Should any nation that fits the definition of "native" (more influence in RegionA than any other region) be able to download it, even if they're not in the region anymore?

Other than "this would be nice" ... anything else worth focusing in on?

Why wouldn't we make it available to everyone?

[Mousebumples]

Bearing in mind that there is no way that I have the tech knowledge to say what is or is not feasible, what sort of access should this sort of thing have?

Should it be restricted to the Founder (and exec-WAD where applicable), where regions that are interested in this sort of thing can semi-occasionally download the RMB Archive for their records? ( similar to backing up your computer every now and again in case of malfunction)

Should any nation that fits the definition of "native" (more influence in RegionA than any other region) be able to download it, even if they're not in the region anymore?

Other than "this would be nice" ... anything else worth focusing in on?

Why wouldn't we make it available to everyone?

Server load issues possibly? Also, why would everyone need to have access to RegionA's RMB posts? It seems like unnecessary work, but - again - I don't know much about coding.

[Riftend]

Why wouldn't we make it available to everyone?

Server load issues possibly? Also, why would everyone need to have access to RegionA's RMB posts? It seems like unnecessary work, but - again - I don't know much about coding.

Well that way everyone can have there own copy if they wish. If it can be done that everyone can have it I don't see why we should restrict it to only certain people.

[Misley]

Why wouldn't we make it available to everyone?

Server load issues possibly? Also, why would everyone need to have access to RegionA's RMB posts? It seems like unnecessary work, but - again - I don't know much about coding.

The server load issue is far more compelling. Anyone can already download an entire region's RMB by downloading the complete Forum View and Region API messages shard. It's not like there's any way to hide RMB posts to show only certain people.

[Bears Armed]

Why wouldn't we make it available to everyone?

Server load issues possibly? Also, why would everyone need to have access to RegionA's RMB posts? It seems like unnecessary work, but - again - I don't know much about coding.

So that anybody who's in the region (or just watching it) when it's on its last legs can aquire a backup copy, just in case? Restricting access to founders & delegates wouldn't really help if the region's reached the point where it no longer has a founder or a delegate, which might well be the case if refounding is seriously under consideration...

[Riftend]

Server load issues possibly? Also, why would everyone need to have access to RegionA's RMB posts? It seems like unnecessary work, but - again - I don't know much about coding.

So that anybody who's in the region (or just watching it) when it's on its last legs can aquire a backup copy, just in case? Restricting access to founders & delegates wouldn't really help if the region's reached the point where it no longer has a founder or a delegate, which might well be the case if refounding is seriously under consideration...

Main reason I was thinking everyone is for the founderless regions - once a raider becomes delegate they can't get a copy~

[Bears Armed]

So that anybody who's in the region (or just watching it) when it's on its last legs can aquire a backup copy, just in case? Restricting access to founders & delegates wouldn't really help if the region's reached the point where it no longer has a founder or a delegate, which might well be the case if refounding is seriously under consideration...

Main reason I was thinking everyone is for the founderless regions - once a raider becomes delegate they can't get a copy~

That too, yes.

[Riftend]

Main reason I was thinking everyone is for the founderless regions - once a raider becomes delegate they can't get a copy~

That too, yes.

Rather than a downloadable RMB - what about an RMB is tied to a regions name. That way f people ever refound they can maintain their previous RMB history?

[Bears Armed]

Rather than a downloadable RMB - what about an RMB is tied to a regions name. That way f people ever refound they can maintain their previous RMB history?

If it's actually technically feasible, which might not be the case bearing in mind how NS has been built-up through the years, then yes.

[Riftend]

Rather than a downloadable RMB - what about an RMB is tied to a regions name. That way f people ever refound they can maintain their previous RMB history?

If it's actually technically feasible, which might not be the case bearing in mind how NS has been built-up through the years, then yes.

Well it makes more sense than being able to save it to your computer. Rather than the RMB be tied to a specific region to have it tied to a Name. Just how I would do it if I worked for NS in the way of coding the game.

[Mousebumples]

Server load issues possibly? Also, why would everyone need to have access to RegionA's RMB posts? It seems like unnecessary work, but - again - I don't know much about coding.

So that anybody who's in the region (or just watching it) when it's on its last legs can aquire a backup copy, just in case? Restricting access to founders & delegates wouldn't really help if the region's reached the point where it no longer has a founder or a delegate, which might well be the case if refounding is seriously under consideration...

Yeah, founders and delegates might be too restrictive, which is why I floated the "Native" option - which is already coded for the poll feature. I think I'm technically still a Native of Monkey Island, even though I'm not in the region right now. I'd expect that would also permit natives that have been ejected by raiders to download a copy, should they so choose.

Can you think of any rational reason why someone outside Natives (and the founder/WAD - in the event that someone from TITO or the UDL wants to backup a copy of the RMB before coordinating a refound) would need to be able to access this sort of thing?

Also, how often should this be updated? If it's in a more readable format (maybe? hopefully? ... requests this non-coder) ... would updating daily with one of the updates be sufficient? I'd expect that having it happen "to the minute" is not feasible due to the frequency with which some RMBs get new posts.

Otherwise, what about having a "Request an RMB Archive" button that would email your nation a copy (to whatever email address you have on your nation's Settings page) when the next RMB Archive is generated?

[Bears Armed]

Can you think of any rational reason why someone outside Natives (and the founder/WAD - in the event that someone from TITO or the UDL wants to backup a copy of the RMB before coordinating a refound) would need to be able to access this sort of thing?

Somebody who used to have a nation in the region, can't take it back there now to re-acquire 'native' status because the (now defunct) founder left a password in place when they CTEd, and wants to refound -- in order to preserve the history -- when the last [inactive] nation present also CTEs?

[Misley]

Can you think of any rational reason why someone outside Natives (and the founder/WAD - in the event that someone from TITO or the UDL wants to backup a copy of the RMB before coordinating a refound) would need to be able to access this sort of thing?

If a historical region that once had RMB activity but now has no active native members and is on the verge of CTEing / is behind a password, it'd be useful for someone outside the region to be able to back that up.

Also, how often should this be updated? If it's in a more readable format (maybe? hopefully? ... requests this non-coder) ... would updating daily with one of the updates be sufficient? I'd expect that having it happen "to the minute" is not feasible due to the frequency with which some RMBs get new posts.

Otherwise, what about having a "Request an RMB Archive" button that would email your nation a copy (to whatever email address you have on your nation's Settings page) when the next RMB Archive is generated?

Is there a reason it can't be generated when the request is made? I don't know how RMB posts are stored on the server, but I would think it shouldn't be too difficult to package the RMB posts upon request. Besides, it's not like all RMBs are going to be archived daily, and there's no sense wasting server resources on an archive that won't be downloaded.

I'm also a bit worried about tying RMB archives to updates, as Ballotonia has said that RMBs are deleted with the region at update. If a request is put in for an empty, password-protected region, it would be too late to save the RMB as it would cease to exist at update.

Maybe if an RMB archive is requested, then that archive is the only archive that will be downloadable until the following major update? That way people can't spam the server with archive requests, but any posts made after the request wouldn't be included.

[Imperializt Russia]

Bearing in mind that there is no way that I have the tech knowledge to say what is or is not feasible, what sort of access should this sort of thing have?

Should it be restricted to the Founder (and exec-WAD where applicable), where regions that are interested in this sort of thing can semi-occasionally download the RMB Archive for their records? ( similar to backing up your computer every now and again in case of malfunction)

Should any nation that fits the definition of "native" (more influence in RegionA than any other region) be able to download it, even if they're not in the region anymore?

Other than "this would be nice" ... anything else worth focusing in on?

The current solution that Sapphire works with at the moment is that our self-appointed, idk, "RP rep" (he wears many hats) simply opens as many posts as he can on page-view (very time consuming to do) and saves the webpage.

It's completely passable so long as the posts are still listed on the server - even the links still work.
However, being able to download some kind of file with this same data would be great. Maybe a bit of code that automatically opens the entire RMB history (in Sapphire this is pushing 75k posts, some of which are quite long RP posts), to save the webpage that way? Personally, I think this would suffice on a basic level for many regions, and have this available as an adminship button only open to founder/delegate, which would aid server load.

Alternatively, some level of "site supporter" could apply to have a region's entire RMB history permanently saved to a special part of the site server, potentially viewable to all users. This may pose bandwidth and server load issues. Again, 75k posts in Sapphire alone, six times the current allowance for forum threads. We certainly don't have the largest RMB history, though we're certainly well up there.
I think Sapphire has at least three site supporters of various levels, and it's "only" $5 for a mid-range supporter badge which I honestly think is worth buying anyway. As such this could possibly be limited to if the delegate/founder hold a site supporter status.

[Shadow Afforess]

NationStates has an API, the API allows access to RMB posts of a region. It should be a small task to build a tool to simply retrieve all RMB messages for a region.

[Registug]

Being the RP rep referred to above, it would be pretty neat to have the option to click a button and then have all the RMB saved. Wouldn't mind paying $5 for that.

dunno what an API is...

[Shadow Afforess]

Being the RP rep referred to above, it would be pretty neat to have the option to click a button and then have all the RMB saved. Wouldn't mind paying $5 for that.

dunno what an API is...

API = Application Programming Interface

API's are a tool to access data directly from the game.

[Misley]

The current solution that Sapphire works with at the moment is that our self-appointed, idk, "RP rep" (he has many titles) simply opens as many posts as he can on page-view (very time consuming to do) and saves the webpage.

It's completely passable so long as the posts are still listed on the server - even the links still work.
However, being able to download some kind of file with this same data would be great. Maybe a bit of code that automatically opens the entire RMB history (in Sapphire this is pushing 75k posts, some of which are quite long RP posts), to save the webpage that way?

If you want to save the entire RMB history to date, do what I did for The Internationale:

• Get Firefox
• Get DownThemAll
• In the DownThemAll Manager, add a batch download for your forum view, which is http://www.nationstates.net/page=displa ... re?start=X. You'll replace "start=X" with "start=[0:Y:25], with "Y" being whatever the "start=" number is for the most recent page on your RMB (which is currently 74750). This will download each forum page, starting with Page 1 (start=0) and stepping up the number by increments of 25 (as each page does) until you reach the last page.
• Start that batch download and wait for it to finish.
• Once the Forum View is downloaded, go to DownThemAll's Preferences > Network and add a Per-Server Limit for nationstates.net at 2 connections and a speed limit of 2 KB/s. This is to keep you under the API request limit for the next step.
• In the DownThemAll Manager, add a batch download for your API messages shard, which is http://www.nationstates.net/cgi-bin/api ... s;offset=X. You'll replace "offset=X" with "offset=[0:Y:10], with "Y" being a multiple of ten that shows the earliest RMB post at the top (which is currently 74760). This will download the entire messages shard, starting with the most recent posts and stepping up by increments of 10 (as only ten messages are shown in the message shard) until you reach the earliest message.
• Start that batch download and wait for it to finish. You might want to go outside or do something else, because this one will take forever.
• Once you have these downloaded, you'll want to strip away everything that doesn't matter. All you want from the Forum view is a list of Post IDs--you'll get the nations, actual post content, and timestamps from the API shards. That's a convoluted process that I can walk you through if you're really interested, but the steps I've shared will at least give you a raw (if ugly) archive.

Keep in mind that if anyone posts to the RMB while you're doing this, especially on the API step, it'll throw you off and you might lose a post, and will be a big pain in the ass to combine the post IDs and API data later. It'll only work if you tell everyone when you're doing it and make sure they don't post on the RMB until it's done.

[Misley]

Which actually raises another point - why is the associated Post ID not delivered in the API request?

[Shadow Afforess]

Which actually raises another point - why is the associated Post ID not delivered in the API request?

Agree, that would be most helpful.

[Sichuan Pepper]

This would certainly be a vast improvement. I would like to see an easy way to retrieve RMB posts. For those not API savvie it would be fantastic if it was a click to download.
I also wonder could the regional history be included? Both are significant losses to regions if they are looking to refound.

[Frisbeeteria]

I suggested to Mousebumples that limiting access and/or timing to such a download would be beneficial in preventing server load issues. My initial thoughts on this were that a Founder or Delegate could download the current contents once every 24 hours.

I realize that the API has much less stringent limits, so perhaps additional frequency should come from API sources rather than a game button. I'm hoping that the proposed in-game button would be formatted nicely to make the choice easier, thus limiting the desire to supplement with API calls. (incidentally, well done on requesting additional information for the API in the appropriate thread.)

We've got ~7000 regions with 2 or more members, so we've got to take that into consideration. We've also got certain chatroom-type RMBs that see 200+ posts a day. We really don't want or need somebody in those regions making a backup after every post. There needs to be a limit.

We should be looking at this choice as if it were a hard disk backup. You don't know when you're going to have a failure, so you back up weekly if you're sensible. It's not something you do after you've been raided or had your region griefed - it's purely preservative maintenance.

From a technical perspective, downloading current contents is relatively easy. Suggestions that an upload facility be included are probably doomed to failure. I'm fairly certain [violet] will back up that statement.