NationStates API (nationdata/regiondata)

[[violet]]

Will it ever be possible to retrieve more than 10 RMB messages at a time? For a script I'm currently developing, I need to retrieve ALL of a region's RMB messages. Retrieving my current region's relatively low number of about 12 thousand RMB pages took 6 hours of non-stop running the script due to the severe limitation of only being able to retrieve 10 RMB messages at a time.

I've added the "limit" parameter to the Regions API "messages" shard, which lets you specify how many messages you want, up to 100.

Please note that the API isn't designed for mass scraping of historical data, like 12,000 pages worth of RMB posts. You are talking about a one-off event, I presume, in which case it doesn't matter how long it takes, and you shouldn't flood the API with requests at the edge of the rate limit.

[Agadar]

Will it ever be possible to retrieve more than 10 RMB messages at a time? For a script I'm currently developing, I need to retrieve ALL of a region's RMB messages. Retrieving my current region's relatively low number of about 12 thousand RMB pages took 6 hours of non-stop running the script due to the severe limitation of only being able to retrieve 10 RMB messages at a time.

I've added the "limit" parameter to the Regions API "messages" shard, which lets you specify how many messages you want, up to 100.

Please note that the API isn't designed for mass scraping of historical data, like 12,000 pages worth of RMB posts. You are talking about a one-off event, I presume, in which case it doesn't matter how long it takes, and you shouldn't flood the API with requests at the edge of the rate limit.

That's great! Even with only 1 request per second (as advised in the documentation when scraping) instead of the normal rate, this will make my script almost 7 times as fast!

To elaborate on my script: it's used for making rankings based on how many posts, average posts per days, likes given, likes received, et cetera everyone in a specified region has. Considering RMB data is unavailable via daily dumps, there is no way for me to retrieve RMB posts other than using the Nationstates API. This script is not designed to be a 'one-off event': I was hoping to release a more definitive version of the script to others at some point, so that they too can enjoy getting such statistics about their own regions. If you want me to halt development on this script, you need only say so.

Considering RMBoards are so massive, would a possible alternative be allowing entire boards to be downloaded in zips, similarly to the daily dumps? The inner workings of Nationstates are a black box to me, but I imagine offering downloads is much less demanding for the server than having to suffer through so many API calls.

[[violet]]

To elaborate on my script: it's used for making rankings based on how many posts, average posts per days, likes given, likes received, et cetera everyone in a specified region has. Considering RMB data is unavailable via daily dumps, there is no way for me to retrieve RMB posts other than using the Nationstates API. This script is not designed to be a 'one-off event': I was hoping to release a more definitive version of the script to others at some point, so that they too can enjoy getting such statistics about their own regions. If you want me to halt development on this script, you need only say so.

No, not at all. I mean 'one-off' in the sense that your script is not (hopefully) attempting to download 120,000 RMB posts every time someone from your region wants to look up their stats. It seems like you only need to do that once.

[Agadar]

To elaborate on my script: it's used for making rankings based on how many posts, average posts per days, likes given, likes received, et cetera everyone in a specified region has. Considering RMB data is unavailable via daily dumps, there is no way for me to retrieve RMB posts other than using the Nationstates API. This script is not designed to be a 'one-off event': I was hoping to release a more definitive version of the script to others at some point, so that they too can enjoy getting such statistics about their own regions. If you want me to halt development on this script, you need only say so.

No, not at all. I mean 'one-off' in the sense that your script is not (hopefully) attempting to download 120,000 RMB posts every time someone from your region wants to look up their stats. It seems like you only need to do that once.

My script currently only downloads to the computer the script is run from; not to a central archive. As such, as it stands, unless region members actively exchange downloaded XML-files containing RMB messages with each other, each member will have to re-download everything individually.

Ideally, these RMB messages would of course be downloaded just once to a central archive, and only updated every now and then to retrieve the last few posts. I'll see what I can do in that regard: maybe I'll end up hosting the archive myself.

Caelapes, I am under the impression you are/have an archive that does more-or-less that?

EDIT: violet, nsarchive.net, which is linked to from the documentation, seems to be dead.

[Eluvatar]

EDIT: violet, nsarchive.net, which is linked to from the documentation, seems to be dead.

Look here.

[Caelapes]

Caelapes, I am under the impression you are/have an archive that does more-or-less that?

Yes, although it's updated much less frequently because of the differing nature of our archive.

The Internationale, before it was refounded (both times, and also including our two refuge regions), saved the entire RMB - both because there's a lot of interesting historical conversation in there, and because most of our regional affairs were handled on-site until very recently, so the RMB had our entire legislative archive and elections process.

We take the XML generated by API calls and insert that data into a SQL database, . I think the same database could be used for your purposes, and would certainly make it very easy to calculate statistics.

Here's the CREATE TABLE query for our archive (some fields you wouldn't need, like "cname" or "region"):

Code: Select all

CREATE TABLE `regional_message_board` (
 `dbid` int(5) unsigned NOT NULL AUTO_INCREMENT,
 `pid` int(8) unsigned NOT NULL DEFAULT '0',
 `timestamp` int(10) unsigned NOT NULL DEFAULT '0',
 `nation` varchar(40) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '',
 `message` text COLLATE utf8mb4_unicode_ci NOT NULL,
 `status` enum('0','1','2','9','10') COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '0' COMMENT '0 = normal; 1 = suppressed; 2 = author deleted; 9 = mod suppressed; 10 = Voice of Mod',
 `suppressor` varchar(40) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `likes` int(3) DEFAULT NULL,
 `likers` varchar(500) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `embassy` varchar(40) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `region` enum('0','1','2') COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '0' COMMENT '0 = The Internationale; 1 = The International; 2 = The Red and Black',
 `cname` varchar(40) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT 'Canonical Nation',
 PRIMARY KEY (`dbid`),
 UNIQUE KEY `pid` (`pid`),
 FULLTEXT KEY `message` (`message`)
) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

Here's the PHP code for inserting into that table:

Code: Select all

<?php
$xml = simplexml_load_file("/path/to/your/rmb/xmlfile.xml", 'SimpleXMLElement', LIBXML_NOCDATA);
$posts = $xml->MESSAGES->POST;

$servername = "mysql.domain.com";
$username = "username";
$password = "password";
$dbname = "database";

$conn = mysqli_connect($servername, $username, $password, $dbname);
if (!$conn) {
   //well shit
   return;
}
mysqli_set_charset($conn, 'utf8mb4');

foreach ($posts as $post) {
   $pid = intval($post->attributes()->id);
   $timestamp = $post->TIMESTAMP;
   $nation = $post->NATION;
   $message = $post->MESSAGE;
   $status = $post->STATUS;
   if ($nation == "NationStates Moderators") {
      $status = 10;
   }
   $likes = $post->LIKES;
   if($post->LIKERS) {
      $likers = "'".$post->LIKERS."'";
   } else {
      $likes = "NULL";
      $likers = "NULL";
   }
   if($post->SUPPRESSOR) {
      $suppressor = "'".$post->SUPPRESSOR."'";
   } else {
      $suppressor = "NULL";
   }
   if($post->EMBASSY) {
      $embassy = "'".$post->EMBASSY."'";
   } else {
      $embassy = "NULL";
   }
   $region = 0; //0 = The Internationale; 1 = The International; 2 = The Red and Black
   
   $sql = "INSERT INTO `regional_message_board` (`pid`, `timestamp`, `nation`, `message`, `status`, `suppressor`, `likes`, `likers`, `embassy`, `region`) VALUES ($pid, $timestamp, '$nation', '$message', '$status', $suppressor, $likes, $likers, $embassy, $region)";
   if ($conn->query($sql) === TRUE) {
      echo "Inserted post $pid.\n";
   } else {
      echo "Error: $sql - ."$conn->error."\n";
   }
}

$conn->close();
?>

I manually escape single quotes by find-and-replacing ' with \' but you could change it to have $message run through mysqli_real_escape_string() before inserting it.

[Agadar]

-snip-

That might prove handy, thanks.

[[violet]]

My script currently only downloads to the computer the script is run from; not to a central archive. As such, as it stands, unless region members actively exchange downloaded XML-files containing RMB messages with each other, each member will have to re-download everything individually.

If this is just to get stats on number of posts & likes, it would be a lot simpler to add API shards that query those numbers directly for you.

[Imperium Anglorum]

If this is just to get stats on number of posts & likes, it would be a lot simpler to add API shards that query those numbers directly for you.

This is a fantastic idea.

[Agadar]

My script currently only downloads to the computer the script is run from; not to a central archive. As such, as it stands, unless region members actively exchange downloaded XML-files containing RMB messages with each other, each member will have to re-download everything individually.

If this is just to get stats on number of posts & likes, it would be a lot simpler to add API shards that query those numbers directly for you.

If you deem the trouble to implement those shards worth it, then sure, sounds like a great idea!

What exactly did you have in mind for the shards? Will they be regional shards that return top-x rankings of a region's RMB statistics? Or will they be nation shards that report on an individual nation's statistics?

For the record: the statistics I'm (currently) after are top 100 ranking lists of:
- nations with most likes received;
- nations with most likes given;
- nations with most posts;
- nations with highest average number of posts per day.

[[violet]]

What exactly did you have in mind for the shards? Will they be regional shards that return top-x rankings of a region's RMB statistics? Or will they be nation shards that report on an individual nation's statistics?

Shards generally get implemented based on what API users want, filtered by what's practical. All of the above are doable, but I like to know that people will actually use something before I build it.

[Agadar]

What exactly did you have in mind for the shards? Will they be regional shards that return top-x rankings of a region's RMB statistics? Or will they be nation shards that report on an individual nation's statistics?

Shards generally get implemented based on what API users want, filtered by what's practical. All of the above are doable, but I like to know that people will actually use something before I build it.

The people in The Western Isles seemed quite fond of the ranking lists I provided, which a regional officer presented in a dispatch (http://www.nationstates.net/page=dispatch/id=667685). I've also already been contacted by a WA delegate from another region, who expressed his interest in the same rankings for his own region. As such, there seems to be some demand for it.

[[violet]]

Wow, that's really nice. Well done.

[[violet]]

I've added the below. These are experimental and undocumented at the moment; please don't smash them with requests, and please provide feedback.

Additions to Region API:

"mostposts" for most frequent posters
"mostliked" for most liked
"mostlikes" for most likes given to other posts

These all take the following optional parameters:

"limit" to specify a maximum number of results
"from" to specify a starting time (in epoch format)
"to" to specify an ending time (in epoch format)

Example: Most liked posts in Testregionia since the start of this month PDT:

https://www.nationstates.net/cgi-bin/ap ... 0;limit=25

[Agadar]

I've added the below. These are experimental and undocumented at the moment; please don't smash them with requests, and please provide feedback.

Additions to Region API:

"mostposts" for most frequent posters
"mostliked" for most liked
"mostlikes" for most likes given to other posts

These all take the following optional parameters:

"limit" to specify a maximum number of results
"from" to specify a starting time (in epoch format)
"to" to specify an ending time (in epoch format)

Example: Most liked posts in Testregionia since the start of this month PDT:

https://www.nationstates.net/cgi-bin/ap ... 0;limit=25

Great! I compared the top 5 results with the results I derived from analyzing the RMB posts in my own code, and they seem to add up nicely. I'll make sure to make a full, in-depth test comparing ALL entries when I'm back home (using the full set of RMB posts from The Western Isles I already downloaded via scraping) and report back here.

One thing to note is that in my script, I calculate the average number of posts per day per nation using the date of the first post of each nation. This doesn't seem possible using the shards currently provided: I can specify a date and retrieve the mostposts-shard, but then the averages-per-day I calculate will be off. For example: a nation that's joined only a week ago and has been consistently posting 10 messages per day will not average 10 messages per day when I calculate the averages using the mostposts shard and the region founded date as starting date, but instead something much lower.

[[violet]]

One thing to note is that in my script, I calculate the average number of posts per day per nation using the date of the first post of each nation. This doesn't seem possible using the shards currently provided: I can specify a date and retrieve the mostposts-shard, but then the averages-per-day I calculate will be off. For example: a nation that's joined only a week ago and has been consistently posting 10 messages per day will not average 10 messages per day when I calculate the averages using the mostposts shard and the region founded date as starting date, but instead something much lower.

I don't think there's an ideal solution for "average posts per time spent in the region" because nothing tracks the latter.

Your current method isn't accurate for a nation that joins the region but doesn't post anything at first. E.g. if I joined a year ago and posted my first message yesterday, it will say I'm averaging 1 post per day.

You can use the Residency score, but it won't work well for nations that leave the region and come back, since it only tracks time spent in the region since the most recent arrival.

[Trotterdam]

I also think that your method would be unfair for nations that join, are active for a bit, go on a hiatus, and then come back and become active again. Such nations would have fewer "average posts per day since joining" than a newer nation that only joined after they came back from their hiatus and is now equally (or even slightly less) active than they are post-hiatus.

I would just calculate how active nations are now, via something like like "average posts in the last month". If you stay inactive long enough, you've essentially quit the region and are no longer relevant, even if you were previously active for years.

[Agadar]

I agree with your assertions.

Violet, I've compared the top 100 rankings returned by the new shards to the rankings I obtained from scraping RMB messages. They match almost exactly, so the algorithm seems valid, or at least when not using the 'from' and 'to' options: I haven't extensively used those options yet, but I'm sure if there are any major bugs with those, we should notice soon enough.

[Imperium Anglorum]

Is there a way to get a sorted list of the most powerful Delegates in the world?

[Eluvatar]

Is there a way to get a sorted list of the most powerful Delegates in the world?

Yes, i.e:

Code: Select all

#!/bin/bash

function dxml() {
    grep "^<$1>" | sed "s/^<$1>\(.\+\)<\/$1>/\1/"
}

function dump_tsv() {
    local -r dump_file="$1"
    shift

    mkdir /tmp/$$-tmpfiles

    ( IFS=$'\t'; echo "${@}" )
    for col in "${@}"; do
        dxml "${col}" < "${dump_file}" > /tmp/$$-tmpfiles/${col}
    done
    pushd /tmp/$$-tmpfiles/ > /dev/null
    paste "${@}"
    popd > /dev/null
    rm -rf /tmp/$$-tmpfiles
}

if [ ! -f regions.xml ]; then
  wget http://www.nationstates.net/pages/regions.xml.gz
  gunzip regions.xml.gz
fi

mkdir /tmp/$$-moretmpfiles
dump_tsv regions.xml DELEGATE DELEGATEVOTES | grep -v '^0\>' > /tmp/$$-moretmpfiles/delegates.tsv
head -n1 /tmp/$$-moretmpfiles/delegates.tsv
tail -n+2 /tmp/$$-moretmpfiles/delegates.tsv | sort -k2 -n -r
rm -rf /tmp/$$-moretmpfiles


Current result looks like:

Code: Select all

DELEGATE DELEGATEVOTES
lord_ravenclaw   1010
louisistan   564
pierconium   557
mousebumples   447
tsunamy   441
elegarth   437
westnesia   322
mcstooley   303
imperium_anglorum   261
starberries   195
solorni   165
libetarian_republics   158
cormactopia_ii   146
the_salaxalans   121
vancouvia   118
mikeswill   115
skothafjordur   104
ceapoyce   98
nonador   82
bearnation   81
the_kerguelen_archipelago   75
celle_franca   73
british_fascist_empire   71
aelbarrow   67
svarttjern   65
acario   61
de_la_malvarma   60
romic   58
damanucus   58
kronomia   55
stavelot-malmedy   54
mons_garle   50
imperial_eagle   47
caelapes   46
parkplace   45
topsail_empire   44
the_irvine_ranch   44
kaiser_adolf   44
atealia   44
independent_federation_of_croatia   40
ainin   39
the_northern_united_american_republic   38
eldrahar   38
scotatrova   37
eisenkernland   37
drifting_colonys_of_aaron   37
ascoobis   37
rain_delay   35
newtexas   35
merlinton   34
ransium   32
blessed_isles   32
ten_cities   31
peallytariall   31
momocarlile   31
earth_defense_force2   31
noladea   30
wikiplay   29
the_rouge_christmas_state   29
topid   28
albarusija   28
victtorya   27
taprobana   27
rundown   27
occoron   27
tigers_of_asia   26
the_german_democratic_reich   26
khanatah   26
ghostopolis   26
united_kingdom_wa_delegate   25
newfound_america   25
vanhania   24
eluvatar   24
corsinthia_13   24
the_city-states_of_kataria   23
sciongrad   23
jeevenaadu   23
vavax   22
the_emerald_world   22
jamaker   22
republic_of_tacos   21
princess_luna   21
hydroponic_nation   21
copenhagen_metropolis   21
the_moravia_ventura   20
qandaristania   20
isaris   20
caffeine_addicts   20
alexiandra   20
yoriz   19
serenarea   19
lamebrainia   19
heku   19
vetega   18
untspah   17
oceanion   17
humpheria_major   17
arkotania   17
xantha   16
...


[HMS Unicorn]

Bug report

I came across this nation: the_will_of_christ. Even though the nation is present in region Europeia, it is not listed in the following API call:

http://www.nationstates.net/cgi-bin/api.cgi?region=europeia&q=nations

It is also not mentioned in either the region or nation dumps.

On the other hand, it does seem to be present in the nation API, e.g.,

http://www.nationstates.net/cgi-bin/api.cgi?nation=the_will_of_christ&q=region

The nation moved from The West Pacific to Europeia during the major update of July 20th. It moved into Europeia after it had finished updating, and it moved out of The West Pacific while it was updating. I do not know whether the nation itself had updated in TWP by the time it moved.

The fact that it seemingly missed an update may be the cause of the bug, though it has been quite a while since then.

[Trotterdam]

That nation seems to be affected by the same bug attributed to a different nation in this thread. It is not specifically an API issue.

[[violet]]

I've added "Private Shards," for when you want to access information that's only available to a logged-in nation.

For now the only shard is "unread", which will tell you how many new TGs, Issues, etc, you have. But I aim to expand it in the future to cover telegrams, issues, and more. So please give this a test and let me know what you think.

For details see "Private Shards" in the API doc!

[Flanderlion]

I've added "Private Shards," for when you want to access information that's only available to a logged-in nation.

For now the only shard is "unread", which will tell you how many new TGs, Issues, etc, you have. But I aim to expand it in the future to cover telegrams, issues, and more. So please give this a test and let me know what you think.

For details see "Private Shards" in the API doc!

Very interesting. The format changes (banners for the titles etc.) look cool, I somehow thought they were spoiler boxes and kept trying to click them, took me a little too long to figure out they weren't. I won't experiment with it properly for a few weeks yet until I have the time to play with my preferred language. I think you've got my code so you know how I do it normally, but using this to view the data, then using the nation pages to actually execute the operations wouldn't be more efficient timewise for me yet, despite the higher rate limits for API.

But for autologin scripts checking TGs and just keeping the nations alive, this seems like it'd speed up the process by a ton. Does this mean the API will add the ability for more actions as well as more information in the future, if you can login to keep nations alive at the faster rate limit than the normal process?

Edit: Split wall of text into paragraphs.

Edit 2: Just realised the advantages of being able to store my passwords not in plain text. Also for just clicking the link (in browser) from my logged in nation, would it be able to appear authenticated so you can view what you'd get for yourself. Obviously for other nations it'd come up with the current 403 error, but for your own, it shouldn't say you don't have permission.

[Caelapes]

Does accessing a private shard update the "Last government activity" text or (edit: I see now that it does do that - and gives a notification when you accidentally use the wrong password. RIP everyone who tries to "hack" into Testlandia by clicking the link in the API documentation ) reset the CTE-by-inactivity timer?