NationStates API (nationdata/regiondata)

[Zemnaya Svoboda]

Not sure if this is really the right place for this, but you should consider having an option for outputting the data in a json format.

That's what I'd do if I were creating the API today. As it is, I don't think there's sufficient demand to migrate, which would take some work.

We don't really have a bandwidth problem; our bottleneck is generating the data to output, regardless of what format we wrap it in.

If it's "generating" it.... does that mean the more gameplayery API shards which are simple lookups of flat data are cheaper than the more roleplayery ones? (I.E. region, wa, endorsements, happenings).

[Holy Trek]

I've made a few changes to the NS API. By which I mean this:

http://nationstates.net/cgi-bin/api.cgi

This has previously been known as regiondata.cgi and nationdata.cgi, both of which continue to work at their old addresses. They essentially forward to this one now.

The API exists so that people can build NS-based sites, such as NSDossier and NSEconomy. We want to support such efforts.

Please visit the API page for information. The major change is that very high rates of queries (more than 100/minute) will result in the IP address being denied service for 15 minutes. This is to ensure regular players don't suffer slowdowns because of overzealous scripts.

In general, scripts should not attempt to look up every resident of a region all at once. This can result in dozens, hundreds, or thousands of near-simultaneous requests to the server, which are difficult or impossible to service. If you need information on large numbers of nations, you should trawl them from the API as a speed-limited background job and store the data locally on your server.

Please let me know if you have any questions or concerns!

Background: This is the result of events documented in the "Scripts banned" thread.

Important Updates! Here and here.

Enlighten me on what, exactly, this is

[Zemnaya Svoboda]

I've made a few changes to the NS API. By which I mean this:

http://nationstates.net/cgi-bin/api.cgi

This has previously been known as regiondata.cgi and nationdata.cgi, both of which continue to work at their old addresses. They essentially forward to this one now.

The API exists so that people can build NS-based sites, such as NSDossier and NSEconomy. We want to support such efforts.

Please visit the API page for information. The major change is that very high rates of queries (more than 100/minute) will result in the IP address being denied service for 15 minutes. This is to ensure regular players don't suffer slowdowns because of overzealous scripts.

In general, scripts should not attempt to look up every resident of a region all at once. This can result in dozens, hundreds, or thousands of near-simultaneous requests to the server, which are difficult or impossible to service. If you need information on large numbers of nations, you should trawl them from the API as a speed-limited background job and store the data locally on your server.

Please let me know if you have any questions or concerns!

Background: This is the result of events documented in the "Scripts banned" thread.

Important Updates! Here and here.

Enlighten me on what, exactly, this is

http://www.nationstates.net/pages/api.html

[Frisbeeteria]

Enlighten me on what, exactly, this is

Scripting tools. If you don't understand what it means, you're not in the target audience.

[Cerberion]

Don't know if this has been requested / rejected.

Can I have a WA nations list in the region API?

[[violet]]

does that mean the more gameplayery API shards which are simple lookups of flat data are cheaper than the more roleplayery ones? (I.E. region, wa, endorsements, happenings).

It's definitely a lot cheaper when we just have to spit out a stored value (like population) than calculate something (like Government Category). You generally won't be able to tell which is which, though.

In any case, all the data we currently supply, we're happy to serve up to whoever wants it. We just want to avoid generating data that has no value. When scripts hit us 50 or 100 times a minute, efficiency is important.

[Zemnaya Svoboda]

does that mean the more gameplayery API shards which are simple lookups of flat data are cheaper than the more roleplayery ones? (I.E. region, wa, endorsements, happenings).

It's definitely a lot cheaper when we just have to spit out a stored value (like population) than calculate something (like Government Category). You generally won't be able to tell which is which, though.

In any case, all the data we currently supply, we're happy to serve up to whoever wants it. We just want to avoid generating data that has no value. When scripts hit us 50 or 100 times a minute, efficiency is important.

On that note, the compare feature lets me compare multiple nations at once, but the API does not.

I suppose though that when displaying a flat field, most of the overhead is loading the nation or region file, so allowing a single API request to reach into multiple files would be unfair.

Still, the way the throttling rules are set up, to scan 15 nation's scores at once using the compare page ten times a minute is far more efficient from my side than to check it via the API one nation at a time 100 times per minute. Obviously I'd use template-overall=none as I always do when fetching HTML from your site, avoiding the check for the 5 latest topics, but it still seems intuitively like this isn't the way you'd like me to get at that info.

[[violet]]

Still, the way the throttling rules are set up, to scan 15 nation's scores at once using the compare page ten times a minute is far more efficient from my side than to check it via the API one nation at a time 100 times per minute. Obviously I'd use template-overall=none as I always do when fetching HTML from your site, avoiding the check for the 5 latest topics, but it still seems intuitively like this isn't the way you'd like me to get at that info.

What info are you gathering? If you want values on a single scale for 2-15 nations, then yeah, I can see that HTML scraping is easier. But the API lets you gather data on lots of scales at once, and at a faster rate, so it's just different implementations of the same thing.

[Poopcannon]

Would it be possible to get a list of the newest nations that only returns recruitable nations, eg ones that are still in the feeders?

As it is, my script has to check each nation on the list until it gets to an acceptable nation, or compare the list against the list of nations in each of the feeders. If I could be certain I can sent them a TG without having to do any checks beforehand, it could hugely reduce my calls to the API.

I'm probably not going to add that, since it would be obsoleted by the new recruiting stuff in the forthcoming TG system. Sorry--I appreciate that the current way is kind of a pain.

Sorry to be a niusance and bring this up again, but since Fris seems convinced that auto-TG programs aren't checking if nations on the newnations list are in feeders, could I appeal this decision and have nations not in the feeders removed from the newnations list?

I don't mind myself, my program is already written to do this, but it would be one apparent problem solved.

[Campinia]

I also think Fris is mistaken, and most if not all scripts currently running do check for this.

[Cerberion]

I'm not sure if my request was overlooked or in some way impossible.

I'm developing a tool that will run some stats on WA nations in my region. As far as I can tell what I will have to do initially is get the regions nation shard, and then step through each nation to discover if they are WA members, and then assess who is endorsing who.

That's going to create a lot of API calls so it would be extremely helpful if I could get a shard added to the regions feed to just call out the WA nations.

If that's not possible or if I have overlooked it, please let me know.

[Zemnaya Svoboda]

I'm not sure if my request was overlooked or in some way impossible.

I'm developing a tool that will run some stats on WA nations in my region. As far as I can tell what I will have to do initially is get the regions nation shard, and then step through each nation to discover if they are WA members, and then assess who is endorsing who.

That's going to create a lot of API calls so it would be extremely helpful if I could get a shard added to the regions feed to just call out the WA nations.

If that's not possible or if I have overlooked it, please let me know.

Use the XML dump http://www.nationstates.net/pages/nations.xml.gz

Still, the way the throttling rules are set up, to scan 15 nation's scores at once using the compare page ten times a minute is far more efficient from my side than to check it via the API one nation at a time 100 times per minute. Obviously I'd use template-overall=none as I always do when fetching HTML from your site, avoiding the check for the 5 latest topics, but it still seems intuitively like this isn't the way you'd like me to get at that info.

What info are you gathering? If you want values on a single scale for 2-15 nations, then yeah, I can see that HTML scraping is easier. But the API lets you gather data on lots of scales at once, and at a faster rate, so it's just different implementations of the same thing.

The Influence score. I don't really care about all the other scores of the non-WA nations in TNP, but the Influence score I want to track.

[Poopcannon]

I also think Fris is mistaken, and most if not all scripts currently running do check for this.

I think so too. It would be ridiculous to assume that every nation in the new nations list is still in a feeder. But if a mod is complaining about it, it may well be a problem...

[Cerberion]

I'm not sure if my request was overlooked or in some way impossible.

I'm developing a tool that will run some stats on WA nations in my region. As far as I can tell what I will have to do initially is get the regions nation shard, and then step through each nation to discover if they are WA members, and then assess who is endorsing who.

That's going to create a lot of API calls so it would be extremely helpful if I could get a shard added to the regions feed to just call out the WA nations.

If that's not possible or if I have overlooked it, please let me know.

Use the XML dump http://www.nationstates.net/pages/nations.xml.gz

The last time I checked the nations XML feed I don't think it had the endorsements shard section. The other issue is that the nations XML feed has a lot of extraneous data that I neither need nor want. I only care about the nations in my region and I can make ~2500 hits on the API faster than I can download and process the 80meg nation XML feed.

[Frisbeeteria]

But if a mod is complaining about it, it may well be a problem

We do get complaints. My best guess is that for every complaint about a mis-sent TG, 99 others are ignored or deleted. Most players never visit the forums and don't know the rules; most long-time puppetmasters can't be bothered to file a GHR about every stupid piece of spam they get.

I also got a complaint yesterday about a new nation that started in one feeder for a day, then moved to a second feeder the next day. Got two sets duplicate TGs from the same recruiters, sending identical 'personalized' messages. That sounds like somebody doing a feeder check but not a new nations check.

Our rules and policies are complex, on both game and script sides. Couple that with potential erratic movement by players, and you have a difficult set of parameters to manage. If you miss a step or two along the way, your recruiting nation will get a warning ... which should be read as an opportunity to correct your assumptions about your script, and fix it.

[Poopcannon]

I'd wonder how someone could make that mistake as well, because it would also be ridiculous to assume that every nation entering a feeder is new and hasn't been TGd...

Maybe I assume too much of people

[Zemnaya Svoboda]

Use the XML dump http://www.nationstates.net/pages/nations.xml.gz

The last time I checked the nations XML feed I don't think it had the endorsements shard section. The other issue is that the nations XML feed has a lot of extraneous data that I neither need nor want. I only care about the nations in my region and I can make ~2500 hits on the API faster than I can download and process the 80meg nation XML feed.

It lists the UNSTATUS of a nation. And if you can't download the 15 megs, un-gzip them, and grab the UNSTATUS from the xml faster than 2500 API requests, something is wrong.

What you do is you get the list of 777 10KI WA nations by munging nations.xml, and then you make 777 API requests.

Downloading nations.xml.gz takes me 10 seconds, and parsing out the list of 10KI WA nations is faster. Let's be conservative and say you have a much slower machine with worse bandwidth. Let's say this takes a whole 2 minutes. Then it's 10 minutes total for this process, vs 25 minutes for the method you're suggesting. In practice though it's going to be 8 minutes.

[Cerberion]

The last time I checked the nations XML feed I don't think it had the endorsements shard section. The other issue is that the nations XML feed has a lot of extraneous data that I neither need nor want. I only care about the nations in my region and I can make ~2500 hits on the API faster than I can download and process the 80meg nation XML feed.

It lists the UNSTATUS of a nation. And if you can't download the 15 megs, un-gzip them, and grab the UNSTATUS from the xml faster than 2500 API requests, something is wrong.

What you do is you get the list of 777 10KI WA nations by munging nations.xml, and then you make 777 API requests.

Downloading nations.xml.gz takes me 10 seconds, and parsing out the list of 10KI WA nations is faster. Let's be conservative and say you have a much slower machine with worse bandwidth. Let's say this takes a whole 2 minutes. Then it's 10 minutes total for this process, vs 25 minutes for the method you're suggesting. In practice though it's going to be 8 minutes.

Well, it could well be something on my implementation, I'm using ruby with the SAX2 parser (I moved to that as REXML couldn't even handle the size of the feed).

It's taking at least ten minutes to strip out the data from the feed and it might be even longer.

Then I have to make the 700+ API calls to see who is endorsing who. which adds on to the annoying amount of time needed to get the data I want.

Hence why I ask if I can just get the WA nations added into the regions xml, because then I can eliminate at least the time needed to parse the nations feed.

It's also more useful to me because I would like to run this script twice a day, and that won't be possible using the daily data dump.

[Ballotonia]

Well, it could well be something on my implementation, I'm using ruby with the SAX2 parser (I moved to that as REXML couldn't even handle the size of the feed).

I ran into that as well. First time I used a plain XML reader to load the nations.xml data it seemed as if the script just sat there and hanged. So I wrote my own little parser which quickly filters out the things I'm interested in. Way way faster. Not as flexible WRT format changes, but the time gain is huge.

It's also more useful to me because I would like to run this script twice a day, and that won't be possible using the daily data dump.

You could still count endorsements twice a day using the same data dump. New WA nations won't have that many endorsements anyway, and you could use the endorsement list of the known WA nations to catch some of the new WA nations when they've endorsed others.

Ballotonia

[Cerberion]

Well, it could well be something on my implementation, I'm using ruby with the SAX2 parser (I moved to that as REXML couldn't even handle the size of the feed).

I ran into that as well. First time I used a plain XML reader to load the nations.xml data it seemed as if the script just sat there and hanged. So I wrote my own little parser which quickly filters out the things I'm interested in. Way way faster. Not as flexible WRT format changes, but the time gain is huge.

It's also more useful to me because I would like to run this script twice a day, and that won't be possible using the daily data dump.

You could still count endorsements twice a day using the same data dump. New WA nations won't have that many endorsements anyway, and you could use the endorsement list of the known WA nations to catch some of the new WA nations when they've endorsed others.

Ballotonia

Well, I'm going to try a different parser. I'll give nokogiri a go and see if I get some performance gains.

It just seems counter productive to have to parse all of the nations when you are only interested in a handful.

[Poopcannon]

There's a lot of data in the dump you wouldn't need to parse. When I load data from it, I run it through some grep and sed calls before parsing it, kind of like Ballotonia suggests, which take about a second. It then takes a tiny fraction of the time to process all the data I'm interested in rather than the minute or two it would otherwise take.

[Zemnaya Svoboda]

I ran into that as well. First time I used a plain XML reader to load the nations.xml data it seemed as if the script just sat there and hanged. So I wrote my own little parser which quickly filters out the things I'm interested in. Way way faster. Not as flexible WRT format changes, but the time gain is huge.



You could still count endorsements twice a day using the same data dump. New WA nations won't have that many endorsements anyway, and you could use the endorsement list of the known WA nations to catch some of the new WA nations when they've endorsed others.

Ballotonia

Well, I'm going to try a different parser. I'll give nokogiri a go and see if I get some performance gains.

It just seems counter productive to have to parse all of the nations when you are only interested in a handful.

Bah I'll show you exactly how to do it :|

Code: Select all

#!/bin/sh

declare -r region=`echo $1 | tr "[:upper:] " "[:lower:]_"`

wget http://www.nationstates.net/pages/nations.xml.gz
gunzip nations.xml.gz

grep "^<NAME>" < nations.xml | sed "s/^<NAME>\(.\+\)<\/NAME>/\1/" > names.txt
grep "^<UNSTATUS>" < nations.xml | sed "s/^<UNSTATUS>\(.\+\)<\/UNSTATUS>/\1/" > uns.txt
grep "^<REGION>" < nations.xml | sed "s/^<REGION>\(.\+\)<\/REGION>/\1/" | tr "[:upper:] " "[:lower:]_" > regions.txt

declare -i first=`grep -n "^${region}$" regions.txt | head -n1 | cut -d: -f1`
declare -i last=`grep -n "^${region}$" regions.txt | tail -n1 | cut -d: -f1`

head -n ${last} names.txt | tail -n+${first} > ${region}-names.txt
head -n ${last} uns.txt | tail -n+${first} > ${region}-uns.txt

exec 3< ${region}-names.txt

while read unstatus
do
  read -u 3 name
  if ! [ "${unstatus}" = "Non-member" ]
  then
    echo ${name}
  fi
done < ${region}-uns.txt

exec 3<&-

The above shell script took 11.369 seconds on my machine to run and identify 928 WA nations in 10000 Islands. I could make it significantly faster if I utilized parallelism, but to keep it simple I chose not to do that at this time. Shaving off a fraction of a second wasn't worth it; nearly the entire time is spent downloading the XML. I could shave off the entire extra second by using a different method of download which can be output directly into a pipe, but still could not shave off more than a second.

If you wish to run the script multiple times per day, what should work sufficiently well is to identify a complete listing once every major update, and when scanning that listing later on also scan any nations endorsing any nation in the list who are not themselves in the list.

It is my understanding that the way regions are currently coded, letting the WA server edit their files could cause serious problems, so a WA nations in <region> API shard does not seem to be exceptionally likely in our near future.

[Cerberion]

Well, I'm going to try a different parser. I'll give nokogiri a go and see if I get some performance gains.

It just seems counter productive to have to parse all of the nations when you are only interested in a handful.

Bah I'll show you exactly how to do it :|

Code: Select all

#!/bin/sh

declare -r region=`echo $1 | tr "[:upper:] " "[:lower:]_"`

wget http://www.nationstates.net/pages/nations.xml.gz
gunzip nations.xml.gz

grep "^<NAME>" < nations.xml | sed "s/^<NAME>\(.\+\)<\/NAME>/\1/" > names.txt
grep "^<UNSTATUS>" < nations.xml | sed "s/^<UNSTATUS>\(.\+\)<\/UNSTATUS>/\1/" > uns.txt
grep "^<REGION>" < nations.xml | sed "s/^<REGION>\(.\+\)<\/REGION>/\1/" | tr "[:upper:] " "[:lower:]_" > regions.txt

declare -i first=`grep -n "^${region}$" regions.txt | head -n1 | cut -d: -f1`
declare -i last=`grep -n "^${region}$" regions.txt | tail -n1 | cut -d: -f1`

head -n ${last} names.txt | tail -n+${first} > ${region}-names.txt
head -n ${last} uns.txt | tail -n+${first} > ${region}-uns.txt

exec 3< ${region}-names.txt

while read unstatus
do
  read -u 3 name
  if ! [ "${unstatus}" = "Non-member" ]
  then
    echo ${name}
  fi
done < ${region}-uns.txt

exec 3<&-

The above shell script took 11.369 seconds on my machine to run and identify 928 WA nations in 10000 Islands. I could make it significantly faster if I utilized parallelism, but to keep it simple I chose not to do that at this time. Shaving off a fraction of a second wasn't worth it; nearly the entire time is spent downloading the XML. I could shave off the entire extra second by using a different method of download which can be output directly into a pipe, but still could not shave off more than a second.

If you wish to run the script multiple times per day, what should work sufficiently well is to identify a complete listing once every major update, and when scanning that listing later on also scan any nations endorsing any nation in the list who are not themselves in the list.

It is my understanding that the way regions are currently coded, letting the WA server edit their files could cause serious problems, so a WA nations in <region> API shard does not seem to be exceptionally likely in our near future.

Well thanks for the example shell script. I guess I hadn't considered taking this approach and had focused on drawing out the data using a full parser which in ruby is bizarrely slow. I'm also unfamiliar with how the data is stored on the server side so I have no idea how simple or complex it is for them to build shards.

I appreciate the feedback.

[Poopcannon]

Sorry to be asking a lot of things. Also, not certain where exactly to ask this.

What is the averaging time for API requests? I assume I can't send 50 requests in 1 second and then none for 29 and get away with it? I want to know for the case where I have to perform several API requests that need to be done together. I'd like to do them quickly, but not so quickly that my script gets banned. (and of course limit my overall rate to 50 per 30 seconds)

[[violet]]

could I appeal this decision and have nations not in the feeders removed from the newnations list?

The 'newnations' list is a list of new nations. What you're requesting is a list of recruitable nations, which would be something different: as I mentioned earlier, I know that would be handy, but won't add it until after we get our new TG system, since otherwise it would immediately need a rewrite.