Archive your RMB as a text file | Python RMB saver v4.1

[Merni]

Now on Github: here. Check the documentation there.

You should have a Python interpreter/IDE installed to use this.

This Python program uses the messages shard of the Region API to get messages from a region's RMB and save them in a (somewhat) neat format in a text file. The data saved are the post author, date and time, post ID, sequential post number (first post 1, etc), number and name of likes, whether the post is suppressed, self-deleted or mod deleted, and if suppressed, by whom, and of course the message content.

Source: download "rmb.py" from Github

Steps to use:

1. Download the source as a .py file with the "download" option on pastebin.
2. Run it by double-clicking, or (preferably) in your IDE.
3. Follow the instructions on the command window. When entering the file name, enter it with full path (using backslashes \ on Windows) and extension (.txt). You can skip entering the path if the file is saved in the same folde as the script (not necessarily the Python folder, please ignore what the script's message says for this). You can use command-line arguments too.

Syntax for CL arguments:

Code: Select all

usage: rmb.py [-h] [-u USERAGENT] [-r REGION] [-o FILE] [-i STARTID]
              [-s STARTNO]

optional arguments:
  -h, --help            show this help message and exit
  -u USERAGENT, --useragent USERAGENT
                        NS API user agent
  -r REGION, --region REGION
                        Region name
  -o FILE, --file FILE  File to output the result (with path if required)
  -i STARTID, --startid STARTID
                        Starting Post ID
  -s STARTNO, --startno STARTNO
                        Starting serial number

If run without the u, r and f arguments it prompts for input.

Running may take some time depending on the size of your RMB. The output screen (console) will print out the number of messages done every 100 messages, so you can keep track. When the program ends, the output window may close on its own. Check your text file to see if the program worked successfully.



If you get an error, please see the spoiler!

If the script stops with an error (starting with "Traceback (most recent call last)" etc on the first line) and the last line of the error contains "invalid token" or something like that, please note the last post ID saved to the text file, start up the script again, enable advanced options, and enter a starting post ID about 100 or 1000 more than that one. (Post IDs are not sequential in a region, so you probably will not miss any posts or maybe a few at most. You may need to advance by a lot if your RMB was inactive at the time of that post.) This error may be caused due to an invalid (control) character in a certain post.

If the output contains ? or weird symbols in unexpected places, please just live with it. This and the above error are caused by inconsistent encoding in the API messages shard.

If you get any other error, or if these fixes don't work, please contact me via TG.

Examples:

--------------------------------------------------------------------------------
20657581: Post 1665 by vavax on 2016-08-04 at 22:42:33 GMT
Likes: 1(merni)

[quote=united_meme_alliance;20656723]NS logic at its finest[/quote]
Damnit Max Barry

where 20657581 is the post ID and 1665 is the sequential number of the post.

--------------------------------------------------------------------------------
19893583: Post 1246 by dragosi on 2016-06-21 at 06:54:32 GMT
Likes: 0()
self-deleted
Message deleted by author

--------------------------------------------------------------------------------
24078428: Post 7532 by please_stay_hydrated on 2017-02-24 at 22:59:44 GMT
Likes: 2(federal_southern_cities:calapsia)
moderator deleted
Message suppressed by a moderator

--------------------------------------------------------------------------------
26640002: Post 11630 by calapsia on 2017-08-04 at 23:03:49 GMT
Likes: 1(new_cheeseland)
suppressed by vavax
[quote=vavax;26631431]Which means I therefore also have positive 0 power.To be fair, this is a real thing that happened.[/quote]
HA. Got you there, dad! Negative zero doesn't exist, therefore neither does your power! :D

Old versions
v1 and v2 require pynationstates installed!
v1 outputs TypeError: 'NoneType' object is not subscriptable when done!
To do:
- Add support for command line arguments so archiving can be auto-scheduled (not that anyone would want to)
- Maybe make quotes more readable / separated from rest of a message (unlikely)
- Exe file rather than py file (unlikelier)
- GUI (unlikeliest)

[Merni]

I have updated the program. Improvements:

1. The program now prints a heading to the text file containing the date/time of the archive, and other info.
2. No more need to edit the code to input region name, user agent etc. Now the program prompts you for those inputs when run.
3. Text file can be saved in any folder, and if it doesn't exist will be created.
4. When the program ends it prints a message and closes the file automatically, instead of throwing an incomprehensible error.
5. Comments in code.
6. Ability to start from a certain post ID, and start from a certain sequential post number in the text file. (Choose Y for advanced options)

[All Wild Things]

Nice work! I haven't tried it, but it's a great idea. I like the idea of being able to easily record an RMB before doing a refound for example.

[Merni]

Nice work! I haven't tried it, but it's a great idea. I like the idea of being able to easily record an RMB before doing a refound for example.

Thanks. I had originally made it as a small thing to archive my region's board in case we were invaded, and I realised that others could be interested as well, so I cleaned it up and put it out here.

I would like to package it so you don't need to install Python IDE and the pynationstates wrapper, but I don't have the time or motivation to figure it out.

[Merni]

I have overhauled the code for v3. Now it uses only pre-installed Python standard library modules, and does not use pynationstates. This means you can just download the script and run it. (as long as you have Python)

[Valentine Z]

I am currently running this on Spyder for one of the regions I am archiving and re-founding. This has been very useful and it works perfectly. Thank you so much! ♥

[LInUxvIllE]

I have updated the program. Improvements:

1. The program now prints a heading to the text file containing the date/time of the archive, and other info.
2. No more need to edit the code to input region name, user agent etc. Now the program prompts you for those inputs when run.
3. Text file can be saved in any folder, and if it doesn't exist will be created.
4. When the program ends it prints a message and closes the file automatically, instead of throwing an incomprehensible error.
5. Comments in code.
6. Ability to start from a certain post ID, and start from a certain sequential post number in the text file. (Choose Y for advanced options)

Is it strictly required to include our nation name/email for this task? When viewing the XML directly through my browser, it doesn't ask for this, just the region name. I am curious because I would like to make a Linux shell script that does what your python program does but without python.

[SherpDaWerp]

I have updated the program. Improvements:

1. The program now prints a heading to the text file containing the date/time of the archive, and other info.
2. No more need to edit the code to input region name, user agent etc. Now the program prompts you for those inputs when run.
3. Text file can be saved in any folder, and if it doesn't exist will be created.
4. When the program ends it prints a message and closes the file automatically, instead of throwing an incomprehensible error.
5. Comments in code.
6. Ability to start from a certain post ID, and start from a certain sequential post number in the text file. (Choose Y for advanced options)

Is it strictly required to include our nation name/email for this task? When viewing the XML directly through my browser, it doesn't ask for this, just the region name. I am curious because I would like to make a Linux shell script that does what your python program does but without python.

If you query the page through the API, you must set a "User-Agent" with some form of identifiable information like nation name or email, so that the API knows who you are and [v] knows who to contact if something goes horribly wrong. Merni's script uses your nation name/email for this user-agent field.

Terms of Use
Generally, you can use the API however you like, and if you try to do something that isn't allowed, your request will simply be denied. For example, there's no punishment for exceeding the rate limit; it simply won't work.

There are, however, a few basic rules you must follow, or we may remove your access to the API or the site in general.

1. Set a User Agent
   You must set your script's UserAgent to something informative, such as the URL of your site or your e-mail address. It can be whatever you want, so long as it allows us to contact you if something goes wrong with your script. If you don't set a UserAgent, you will receive response status 403 "Forbidden" and some explanatory text.

If you've accessed the page via your browser then you've provided the default browser user-agent (for instance, my default browser user-agent is "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0") This is enough to get around the no-useragent filter described, but it's strongly suggested that if you make a script you set the user agent to something a lot more informative.

You can check your user agent here: https://www.nationstates.net/cgi-bin/api.cgi?a=useragent If you intend to go the API route with your shell script then you must also follow ratelimits and other rules - check the documentation (linked as the author of the above quote) for the full details.

Happy Scripting!

[LInUxvIllE]

Is it strictly required to include our nation name/email for this task? When viewing the XML directly through my browser, it doesn't ask for this, just the region name. I am curious because I would like to make a Linux shell script that does what your python program does but without python.

If you query the page through the API, you must set a "User-Agent" with some form of identifiable information like nation name or email, so that the API knows who you are and [v] knows who to contact if something goes horribly wrong. Merni's script uses your nation name/email for this user-agent field.

Terms of Use
Generally, you can use the API however you like, and if you try to do something that isn't allowed, your request will simply be denied. For example, there's no punishment for exceeding the rate limit; it simply won't work.

There are, however, a few basic rules you must follow, or we may remove your access to the API or the site in general.

1. Set a User Agent
   You must set your script's UserAgent to something informative, such as the URL of your site or your e-mail address. It can be whatever you want, so long as it allows us to contact you if something goes wrong with your script. If you don't set a UserAgent, you will receive response status 403 "Forbidden" and some explanatory text.

If you've accessed the page via your browser then you've provided the default browser user-agent (for instance, my default browser user-agent is "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0") This is enough to get around the no-useragent filter described, but it's strongly suggested that if you make a script you set the user agent to something a lot more informative.

You can check your user agent here: https://www.nationstates.net/cgi-bin/api.cgi?a=useragent If you intend to go the API route with your shell script then you must also follow ratelimits and other rules - check the documentation (linked as the author of the above quote) for the full details.

Happy Scripting!

Thank you! I am aware of the rules in the documentation. As far as a useragent, would something like this suffice:

Code: Select all

```bash
export USERAGENT='email:<email>,nation:<nation>,purpose:keeping_updated_RMB_archive'
```


[Merni]

If you query the page through the API, you must set a "User-Agent" with some form of identifiable information like nation name or email, so that the API knows who you are and [v] knows who to contact if something goes horribly wrong. Merni's script uses your nation name/email for this user-agent field.



If you've accessed the page via your browser then you've provided the default browser user-agent (for instance, my default browser user-agent is "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0") This is enough to get around the no-useragent filter described, but it's strongly suggested that if you make a script you set the user agent to something a lot more informative.

You can check your user agent here: https://www.nationstates.net/cgi-bin/api.cgi?a=useragent If you intend to go the API route with your shell script then you must also follow ratelimits and other rules - check the documentation (linked as the author of the above quote) for the full details.

Happy Scripting!

Thank you! I am aware of the rules in the documentation. As far as a useragent, would something like this suffice:

Code: Select all

```bash
export USERAGENT='email:<email>,nation:<nation>,purpose:keeping_updated_RMB_archive'
```


That should be more than enough. (set the useragent in the header "User-Agent")

[SherpDaWerp]

command-line scripting is very much not my specialty, but NS is picky about the name of the field they accept as useragent.
The site only accepts "User-Agent", not "UserAgent, "user-agent", "useragent", "USERAGENT" or any permutation thereof.

You have to send it as a http/https header when you make the request to the API.

[LInUxvIllE]

command-line scripting is very much not my specialty, but NS is picky about the name of the field they accept as useragent.
The site only accepts "User-Agent", not "UserAgent, "user-agent", "useragent", "USERAGENT" or any permutation thereof.

You have to send it as a http/https header when you make the request to the API.

`USERAGENT` would just be a variable defined in the shell script but I might do that in Python if it makes sending the HTTP header easier.

[SherpDaWerp]

command-line scripting is very much not my specialty, but NS is picky about the name of the field they accept as useragent.
The site only accepts "User-Agent", not "UserAgent, "user-agent", "useragent", "USERAGENT" or any permutation thereof.

You have to send it as a http/https header when you make the request to the API.

`USERAGENT` would just be a variable defined in the shell script but I might do that in Python if it makes sending the HTTP header easier.

Python, I can do. The requests library makes API pretty easy - you pass headers as an argument along with url with requests.get. I'm in the midst of making a wrapper that does all of this on it's own, but it's not ready for release yet. There are other python NS API wrappers out there, anyway.

Here's some snippets of how I define headers and then "get" the data. Obviously the class variables (self.userAgent and self.auth) change based on the instance. If you're just archiving an RMB, you don't need any of the authentication fields.

Code: Select all

headers = {'User-Agent': self.userAgent,
'X-Password': self.auth[0],
'X-Autologin': self.auth[1],
'X-Pin': self.auth[2]}

data = requests.get(url, headers=headers)

resp_header = data.headers
resp_data = xmltodict.parse(data.content, "utf-8")

In any case, the thread of a different python RMB saver tool is hardly the best place to have this discussion - if you've got any more questions TG me.

[Merni]

The script (v4) now supports command-line arguments for input.

[Merni]

Update
I didn't think this would ever happen, but the script now has a graphical front-end!
To download: Click here
How to use the GUI: Click here
More info on both versions: here and here.


Also for Windows users, I made Windows executables (.exe) of the Python script. This means that Windows users don't need to install Python to use the script any more. See here.

Edit: Non-Windows users, or even Windows users with Python installed, can use the old method, which has not changed. Download the source code from GitHub and use the "rmb.py" file, which is the command-line tool.

Technical notes:
The GUI was made in Visual Basic 6.0 (from 1998!) the only way I know to make GUIs. Hence, it only works on Windows. It also looks retro (read: terrible). Behind the scenes, the way it works is by taking the input from the user, then calling the Python script using command-line arguments based on the input. (You can read the source on Github)

The GUI hasn't been extensively tested, please send me a TG if anything goes wrong.