NATION

PASSWORD

Archive your RMB as a text file | Python RMB saver v4

Bug reports, general help, ideas for improvements, and questions about how things are meant to work.
User avatar
Merni
Ambassador
 
Posts: 1133
Founded: May 03, 2016
Democratic Socialists

Archive your RMB as a text file | Python RMB saver v4

Postby Merni » Sun Dec 01, 2019 2:41 am

Now on Github: here
You should have a Python interpreter/IDE installed to use this.


This Python program uses the messages shard of the Region API to get messages from a region's RMB and save them in a (somewhat) neat format in a text file. The data saved are the post author, date and time, post ID, sequential post number (first post 1, etc), number and name of likes, whether the post is suppressed, self-deleted or mod deleted, and if suppressed, by whom, and of course the message content.

Source: download "rmb.py" from Github

Steps to use:
  1. Download the source as a .py file with the "download" option on pastebin.
  2. Run it by double-clicking, or (preferably) in your IDE.
  3. Follow the instructions on the command window. When entering the file name, enter it with full path (using backslashes \ on Windows) and extension (.txt). You can skip entering the path if the file is saved in the same folde as the script (not necessarily the Python folder, please ignore what the script's message says for this). You can use command-line arguments too.

Syntax for CL arguments:
Code: Select all
usage: rmb.py [-h] [-u USERAGENT] [-r REGION] [-o FILE] [-i STARTID]
              [-s STARTNO]

optional arguments:
  -h, --help            show this help message and exit
  -u USERAGENT, --useragent USERAGENT
                        NS API user agent
  -r REGION, --region REGION
                        Region name
  -o FILE, --file FILE  File to output the result (with path if required)
  -i STARTID, --startid STARTID
                        Starting Post ID
  -s STARTNO, --startno STARTNO
                        Starting serial number


If run without the u, r and f arguments it prompts for input.

Running may take some time depending on the size of your RMB. The output screen (console) will print out the number of messages done every 100 messages, so you can keep track. When the program ends, the output window may close on its own. Check your text file to see if the program worked successfully.



If you get an error, please see the spoiler!
If the script stops with an error (starting with "Traceback (most recent call last)" etc on the first line) and the last line of the error contains "invalid token" or something like that, please note the last post ID saved to the text file, start up the script again, enable advanced options, and enter a starting post ID about 100 or 1000 more than that one. (Post IDs are not sequential in a region, so you probably will not miss any posts or maybe a few at most. You may need to advance by a lot if your RMB was inactive at the time of that post.) This error may be caused due to an invalid (control) character in a certain post.

If the output contains ? or weird symbols in unexpected places, please just live with it. This and the above error are caused by inconsistent encoding in the API messages shard.

If you get any other error, or if these fixes don't work, please contact me via TG.

Examples:
--------------------------------------------------------------------------------
20657581: Post 1665 by vavax on 2016-08-04 at 22:42:33 GMT
Likes: 1(merni)

[quote=united_meme_alliance;20656723]NS logic at its finest[/quote]
Damnit Max Barry

where 20657581 is the post ID and 1665 is the sequential number of the post.
--------------------------------------------------------------------------------
19893583: Post 1246 by dragosi on 2016-06-21 at 06:54:32 GMT
Likes: 0()
self-deleted
Message deleted by author

--------------------------------------------------------------------------------
24078428: Post 7532 by please_stay_hydrated on 2017-02-24 at 22:59:44 GMT
Likes: 2(federal_southern_cities:calapsia)
moderator deleted
Message suppressed by a moderator

--------------------------------------------------------------------------------
26640002: Post 11630 by calapsia on 2017-08-04 at 23:03:49 GMT
Likes: 1(new_cheeseland)
suppressed by vavax
[quote=vavax;26631431]Which means I therefore also have positive 0 power.To be fair, this is a real thing that happened.[/quote]
HA. Got you there, dad! Negative zero doesn't exist, therefore neither does your power! :D

[spoiler=Old versions and to do list
v1 and v2 require pynationstates installed!
v1 outputs TypeError: 'NoneType' object is not subscriptable when done!
To do:
- Add support for command line arguments so archiving can be auto-scheduled (not that anyone would want to)
- Maybe make quotes more readable / separated from rest of a message (unlikely)
- Exe file rather than py file (unlikelier)
- GUI (unlikeliest)
[/spoiler]
Last edited by Merni on Sat Jun 13, 2020 4:57 am, edited 13 times in total.
COVID19 still exists! | Free time? Help Gutenberg!
The Labyrinth | GA Committee List | Tools written by me | NS Parliament 2: Zhang Xiaoqing
I've noticed that everybody that is for abortion has already been born. — Ronald Reagan

When the people are being beaten with a stick, they are not much happier if it is called 'the People’s Stick.' — Mikhail Bakunin (to Karl Marx)

You're supposed to be employing the arts of diplomacy, not the ruddy great thumping sledgehammers of diplomacy. — Ardchoille

The West won the world not by the superiority of its ideas or values or religion [...] but rather by its superiority in applying organised violence. — Samuel P. Huntington (even he said that!)

Compass E:-7.75 S:-3.64

User avatar
Merni
Ambassador
 
Posts: 1133
Founded: May 03, 2016
Democratic Socialists

Postby Merni » Mon Dec 09, 2019 5:23 am

I have updated the program. Improvements:
  1. The program now prints a heading to the text file containing the date/time of the archive, and other info.
  2. No more need to edit the code to input region name, user agent etc. Now the program prompts you for those inputs when run.
  3. Text file can be saved in any folder, and if it doesn't exist will be created.
  4. When the program ends it prints a message and closes the file automatically, instead of throwing an incomprehensible error.
  5. Comments in code.
  6. Ability to start from a certain post ID, and start from a certain sequential post number in the text file. (Choose Y for advanced options)
Last edited by Merni on Mon Dec 09, 2019 8:06 am, edited 1 time in total.
COVID19 still exists! | Free time? Help Gutenberg!
The Labyrinth | GA Committee List | Tools written by me | NS Parliament 2: Zhang Xiaoqing
I've noticed that everybody that is for abortion has already been born. — Ronald Reagan

When the people are being beaten with a stick, they are not much happier if it is called 'the People’s Stick.' — Mikhail Bakunin (to Karl Marx)

You're supposed to be employing the arts of diplomacy, not the ruddy great thumping sledgehammers of diplomacy. — Ardchoille

The West won the world not by the superiority of its ideas or values or religion [...] but rather by its superiority in applying organised violence. — Samuel P. Huntington (even he said that!)

Compass E:-7.75 S:-3.64

User avatar
All Wild Things
Spokesperson
 
Posts: 190
Founded: Apr 24, 2017
Civil Rights Lovefest

Postby All Wild Things » Wed Dec 11, 2019 1:13 am

Nice work! I haven't tried it, but it's a great idea. I like the idea of being able to easily record an RMB before doing a refound for example.

User avatar
Merni
Ambassador
 
Posts: 1133
Founded: May 03, 2016
Democratic Socialists

Postby Merni » Wed Dec 11, 2019 8:58 am

All Wild Things wrote:Nice work! I haven't tried it, but it's a great idea. I like the idea of being able to easily record an RMB before doing a refound for example.

Thanks. I had originally made it as a small thing to archive my region's board in case we were invaded, and I realised that others could be interested as well, so I cleaned it up and put it out here.

I would like to package it so you don't need to install Python IDE and the pynationstates wrapper, but I don't have the time or motivation to figure it out.
Last edited by Merni on Wed Dec 11, 2019 9:00 am, edited 1 time in total.
COVID19 still exists! | Free time? Help Gutenberg!
The Labyrinth | GA Committee List | Tools written by me | NS Parliament 2: Zhang Xiaoqing
I've noticed that everybody that is for abortion has already been born. — Ronald Reagan

When the people are being beaten with a stick, they are not much happier if it is called 'the People’s Stick.' — Mikhail Bakunin (to Karl Marx)

You're supposed to be employing the arts of diplomacy, not the ruddy great thumping sledgehammers of diplomacy. — Ardchoille

The West won the world not by the superiority of its ideas or values or religion [...] but rather by its superiority in applying organised violence. — Samuel P. Huntington (even he said that!)

Compass E:-7.75 S:-3.64

User avatar
Merni
Ambassador
 
Posts: 1133
Founded: May 03, 2016
Democratic Socialists

Postby Merni » Sun Dec 29, 2019 10:29 pm

I have overhauled the code for v3. Now it uses only pre-installed Python standard library modules, and does not use pynationstates. This means you can just download the script and run it. (as long as you have Python)
COVID19 still exists! | Free time? Help Gutenberg!
The Labyrinth | GA Committee List | Tools written by me | NS Parliament 2: Zhang Xiaoqing
I've noticed that everybody that is for abortion has already been born. — Ronald Reagan

When the people are being beaten with a stick, they are not much happier if it is called 'the People’s Stick.' — Mikhail Bakunin (to Karl Marx)

You're supposed to be employing the arts of diplomacy, not the ruddy great thumping sledgehammers of diplomacy. — Ardchoille

The West won the world not by the superiority of its ideas or values or religion [...] but rather by its superiority in applying organised violence. — Samuel P. Huntington (even he said that!)

Compass E:-7.75 S:-3.64

User avatar
Valentine Z
Powerbroker
 
Posts: 7804
Founded: Nov 08, 2015
Scandinavian Liberal Paradise

Postby Valentine Z » Sat Jan 18, 2020 9:03 am

I am currently running this on Spyder for one of the regions I am archiving and re-founding. This has been very useful and it works perfectly. Thank you so much! ♥



Valentine Z GIF Flag (Improved!)
The Valentine Z CV.
NationStates 1st Flag Bracket - 1st place!
Issues Thread 4th in Gen Sec

The Sixty! Opinions Deposit! Valentian Stories! Gwen's Adventures! Cat Anthem! Valentian News.
If you are reading my sig, I want you to have the best day ever ! You are worth it, do not let anyone get you down !
Glory to De Geweldige Sierlijke Katachtige Utopia en Zijne Autonome Machten ov Valentine Z !
Q & A Here! | Heights of NS! Add yours now!

Clarissa mistaken for Smurf/Avatar: 10
Valentijn called She: 38
Disappointing F7 Responses: 197

• Eat my freaking tranquility. - Zenyatta
• Never trouble trouble until trouble troubles you.
• World Map is a cat playing with Australia.

• Am I a pretty girl?

User avatar
Linuxville
Lobbyist
 
Posts: 14
Founded: May 13, 2019
Democratic Socialists

Postby Linuxville » Wed Apr 29, 2020 11:22 pm

Merni wrote:I have updated the program. Improvements:
  1. The program now prints a heading to the text file containing the date/time of the archive, and other info.
  2. No more need to edit the code to input region name, user agent etc. Now the program prompts you for those inputs when run.
  3. Text file can be saved in any folder, and if it doesn't exist will be created.
  4. When the program ends it prints a message and closes the file automatically, instead of throwing an incomprehensible error.
  5. Comments in code.
  6. Ability to start from a certain post ID, and start from a certain sequential post number in the text file. (Choose Y for advanced options)


Is it strictly required to include our nation name/email for this task? When viewing the XML directly through my browser, it doesn't ask for this, just the region name. I am curious because I would like to make a Linux shell script that does what your python program does but without python.

User avatar
SherpDaWerp
Diplomat
 
Posts: 626
Founded: Mar 02, 2016
Civil Rights Lovefest

Postby SherpDaWerp » Wed Apr 29, 2020 11:48 pm

Linuxville wrote:
Merni wrote:I have updated the program. Improvements:
  1. The program now prints a heading to the text file containing the date/time of the archive, and other info.
  2. No more need to edit the code to input region name, user agent etc. Now the program prompts you for those inputs when run.
  3. Text file can be saved in any folder, and if it doesn't exist will be created.
  4. When the program ends it prints a message and closes the file automatically, instead of throwing an incomprehensible error.
  5. Comments in code.
  6. Ability to start from a certain post ID, and start from a certain sequential post number in the text file. (Choose Y for advanced options)


Is it strictly required to include our nation name/email for this task? When viewing the XML directly through my browser, it doesn't ask for this, just the region name. I am curious because I would like to make a Linux shell script that does what your python program does but without python.

If you query the page through the API, you must set a "User-Agent" with some form of identifiable information like nation name or email, so that the API knows who you are and [v] knows who to contact if something goes horribly wrong. Merni's script uses your nation name/email for this user-agent field.

The API Documentation wrote:Terms of Use
Generally, you can use the API however you like, and if you try to do something that isn't allowed, your request will simply be denied. For example, there's no punishment for exceeding the rate limit; it simply won't work.

There are, however, a few basic rules you must follow, or we may remove your access to the API or the site in general.
  1. Set a User Agent
    You must set your script's UserAgent to something informative, such as the URL of your site or your e-mail address. It can be whatever you want, so long as it allows us to contact you if something goes wrong with your script. If you don't set a UserAgent, you will receive response status 403 "Forbidden" and some explanatory text.


If you've accessed the page via your browser then you've provided the default browser user-agent (for instance, my default browser user-agent is "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0") This is enough to get around the no-useragent filter described, but it's strongly suggested that if you make a script you set the user agent to something a lot more informative.

You can check your user agent here: https://www.nationstates.net/cgi-bin/api.cgi?a=useragent If you intend to go the API route with your shell script then you must also follow ratelimits and other rules - check the documentation (linked as the author of the above quote) for the full details.

Happy Scripting!
IRL: High school student who actually doesn't have any assignments left to do...
NS: Owner of Imperial Stronghold | Citizen of The North Pacific | Writer of Issues | One-time Delegate of Glass Gallows | My color is epic

User avatar
Linuxville
Lobbyist
 
Posts: 14
Founded: May 13, 2019
Democratic Socialists

Postby Linuxville » Thu Apr 30, 2020 1:10 am

SherpDaWerp wrote:
Linuxville wrote:
Is it strictly required to include our nation name/email for this task? When viewing the XML directly through my browser, it doesn't ask for this, just the region name. I am curious because I would like to make a Linux shell script that does what your python program does but without python.

If you query the page through the API, you must set a "User-Agent" with some form of identifiable information like nation name or email, so that the API knows who you are and [v] knows who to contact if something goes horribly wrong. Merni's script uses your nation name/email for this user-agent field.

The API Documentation wrote:Terms of Use
Generally, you can use the API however you like, and if you try to do something that isn't allowed, your request will simply be denied. For example, there's no punishment for exceeding the rate limit; it simply won't work.

There are, however, a few basic rules you must follow, or we may remove your access to the API or the site in general.
  1. Set a User Agent
    You must set your script's UserAgent to something informative, such as the URL of your site or your e-mail address. It can be whatever you want, so long as it allows us to contact you if something goes wrong with your script. If you don't set a UserAgent, you will receive response status 403 "Forbidden" and some explanatory text.


If you've accessed the page via your browser then you've provided the default browser user-agent (for instance, my default browser user-agent is "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0") This is enough to get around the no-useragent filter described, but it's strongly suggested that if you make a script you set the user agent to something a lot more informative.

You can check your user agent here: https://www.nationstates.net/cgi-bin/api.cgi?a=useragent If you intend to go the API route with your shell script then you must also follow ratelimits and other rules - check the documentation (linked as the author of the above quote) for the full details.

Happy Scripting!

Thank you! I am aware of the rules in the documentation. As far as a useragent, would something like this suffice:
Code: Select all
```bash
export USERAGENT='email:<email>,nation:<nation>,purpose:keeping_updated_RMB_archive'
```

User avatar
Merni
Ambassador
 
Posts: 1133
Founded: May 03, 2016
Democratic Socialists

Postby Merni » Thu Apr 30, 2020 1:16 am

Linuxville wrote:
SherpDaWerp wrote:If you query the page through the API, you must set a "User-Agent" with some form of identifiable information like nation name or email, so that the API knows who you are and [v] knows who to contact if something goes horribly wrong. Merni's script uses your nation name/email for this user-agent field.



If you've accessed the page via your browser then you've provided the default browser user-agent (for instance, my default browser user-agent is "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0") This is enough to get around the no-useragent filter described, but it's strongly suggested that if you make a script you set the user agent to something a lot more informative.

You can check your user agent here: https://www.nationstates.net/cgi-bin/api.cgi?a=useragent If you intend to go the API route with your shell script then you must also follow ratelimits and other rules - check the documentation (linked as the author of the above quote) for the full details.

Happy Scripting!

Thank you! I am aware of the rules in the documentation. As far as a useragent, would something like this suffice:
Code: Select all
```bash
export USERAGENT='email:<email>,nation:<nation>,purpose:keeping_updated_RMB_archive'
```

That should be more than enough. (set the useragent in the header "User-Agent")
COVID19 still exists! | Free time? Help Gutenberg!
The Labyrinth | GA Committee List | Tools written by me | NS Parliament 2: Zhang Xiaoqing
I've noticed that everybody that is for abortion has already been born. — Ronald Reagan

When the people are being beaten with a stick, they are not much happier if it is called 'the People’s Stick.' — Mikhail Bakunin (to Karl Marx)

You're supposed to be employing the arts of diplomacy, not the ruddy great thumping sledgehammers of diplomacy. — Ardchoille

The West won the world not by the superiority of its ideas or values or religion [...] but rather by its superiority in applying organised violence. — Samuel P. Huntington (even he said that!)

Compass E:-7.75 S:-3.64

User avatar
SherpDaWerp
Diplomat
 
Posts: 626
Founded: Mar 02, 2016
Civil Rights Lovefest

Postby SherpDaWerp » Thu Apr 30, 2020 1:21 am

command-line scripting is very much not my specialty, but NS is picky about the name of the field they accept as useragent.
The site only accepts "User-Agent", not "UserAgent, "user-agent", "useragent", "USERAGENT" or any permutation thereof.

You have to send it as a http/https header when you make the request to the API.
IRL: High school student who actually doesn't have any assignments left to do...
NS: Owner of Imperial Stronghold | Citizen of The North Pacific | Writer of Issues | One-time Delegate of Glass Gallows | My color is epic

User avatar
Linuxville
Lobbyist
 
Posts: 14
Founded: May 13, 2019
Democratic Socialists

Postby Linuxville » Thu Apr 30, 2020 1:31 am

SherpDaWerp wrote:command-line scripting is very much not my specialty, but NS is picky about the name of the field they accept as useragent.
The site only accepts "User-Agent", not "UserAgent, "user-agent", "useragent", "USERAGENT" or any permutation thereof.

You have to send it as a http/https header when you make the request to the API.

`USERAGENT` would just be a variable defined in the shell script but I might do that in Python if it makes sending the HTTP header easier.

User avatar
SherpDaWerp
Diplomat
 
Posts: 626
Founded: Mar 02, 2016
Civil Rights Lovefest

Postby SherpDaWerp » Thu Apr 30, 2020 1:51 am

Linuxville wrote:
SherpDaWerp wrote:command-line scripting is very much not my specialty, but NS is picky about the name of the field they accept as useragent.
The site only accepts "User-Agent", not "UserAgent, "user-agent", "useragent", "USERAGENT" or any permutation thereof.

You have to send it as a http/https header when you make the request to the API.

`USERAGENT` would just be a variable defined in the shell script but I might do that in Python if it makes sending the HTTP header easier.

Python, I can do. The requests library makes API pretty easy - you pass headers as an argument along with url with requests.get. I'm in the midst of making a wrapper that does all of this on it's own, but it's not ready for release yet. There are other python NS API wrappers out there, anyway.

Here's some snippets of how I define headers and then "get" the data. Obviously the class variables (self.userAgent and self.auth) change based on the instance. If you're just archiving an RMB, you don't need any of the authentication fields.
Code: Select all
headers = {'User-Agent': self.userAgent,
'X-Password': self.auth[0],
'X-Autologin': self.auth[1],
'X-Pin': self.auth[2]}

data = requests.get(url, headers=headers)

resp_header = data.headers
resp_data = xmltodict.parse(data.content, "utf-8")


In any case, the thread of a different python RMB saver tool is hardly the best place to have this discussion - if you've got any more questions TG me.
IRL: High school student who actually doesn't have any assignments left to do...
NS: Owner of Imperial Stronghold | Citizen of The North Pacific | Writer of Issues | One-time Delegate of Glass Gallows | My color is epic

User avatar
Merni
Ambassador
 
Posts: 1133
Founded: May 03, 2016
Democratic Socialists

Postby Merni » Sat Jun 13, 2020 4:55 am

The script (v4) now supports command-line arguments for input.
COVID19 still exists! | Free time? Help Gutenberg!
The Labyrinth | GA Committee List | Tools written by me | NS Parliament 2: Zhang Xiaoqing
I've noticed that everybody that is for abortion has already been born. — Ronald Reagan

When the people are being beaten with a stick, they are not much happier if it is called 'the People’s Stick.' — Mikhail Bakunin (to Karl Marx)

You're supposed to be employing the arts of diplomacy, not the ruddy great thumping sledgehammers of diplomacy. — Ardchoille

The West won the world not by the superiority of its ideas or values or religion [...] but rather by its superiority in applying organised violence. — Samuel P. Huntington (even he said that!)

Compass E:-7.75 S:-3.64


Advertisement

Remove ads

Return to Technical

Who is online

Users browsing this forum: Destructive Government Economic System, Preusses, Sebero Sree

Advertisement

Remove ads