NationStates API (nationdata/regiondata)

[[violet]]

I've made a few changes to the NS API. By which I mean this:

http://www.nationstates.net/pages/api.html

This has previously been known as regiondata.cgi and nationdata.cgi, both of which continue to work at their old addresses. They essentially forward to this one now. which have been retired as of August 2013.

The API exists so that people can build NS-based sites, such as NSDossier and NSEconomy. We want to support such efforts.

Please visit the API documentation page for information. The major change is that very high rates of queries (more than 100/minute) will result in the IP address being denied service for 15 minutes. This is to ensure regular players don't suffer slowdowns because of overzealous scripts.

In general, scripts should not attempt to look up every resident of a region all at once. This can result in dozens, hundreds, or thousands of near-simultaneous requests to the server, which are difficult or impossible to service. If you need information on large numbers of nations, you should trawl them from the API as a speed-limited background job and store the data locally on your server.

Please let me know if you have any questions or concerns!

Background: This is the result of events documented in the "Scripts banned" thread.

[NewTexas]

That is great. A very important characteristic of an API is details. Now that you are encouraging developers to use a database to store the data versus just hitting the feeds any time they want it, some details are needed. In particular, information is needed about the data such as type and length. When getting data from the feeds, it did not matter because it was never being stored anywhere. When creating a database, you need to know how what kind of things you are going to put in it and how big they are possibly going to be.

Here are some guesses we have put together, if you could confirm and/or correct the following information, that would be great:

Region Data
Name Data Type

Code: Select all

NAME   VARCHAR(30)
FACTBOOK   NVARCHAR(800)
NUMNATIONS   INTEGER
NATIONS   BLOB
DELEGATE   VARCHAR(22)
DELEGATEVOTES   INTEGER
FOUNDER   VARCHAR(22)
POWER   VARCHAR(30)
FLAG   VARCHAR(50)

Nation Data
Name Data Type

Code: Select all

NAME   VARCHAR(22)
TYPE   VARCHAR(40)
FULLNAME   VARCHAR(63)
MOTTO   NVARCHAR(125)
CATEGORY   VARCHAR(25)
UNSTATUS   CHAR(2)
CIVILRIGHTS   VARCHAR(25)
ECONOMY   VARCHAR(25)
POLITICALFREEDOM   VARCHAR(25)
REGION   VARCHAR(30)
POPULATION   INTEGER
TAX   INTEGER
ANIMAL   VARCHAR(25)
CURRENCY   VARCHAR(25)
FLAG   VARCHAR(50)
MAJORINDUSTRY   VARCHAR(50)
GOVTPRIORITY   VARCHAR(50)
ADMINISTRATION   CHAR(4)
WELFARE   CHAR(4)
HEALTHCARE   CHAR(4)
EDUCATION   CHAR(4)
SPIRITUALITY   CHAR(4)
DEFENCE   CHAR(4)
LAWANDORDER   CHAR(4)
COMMERCE   CHAR(4)
PUBLICTRANSPORT   CHAR(4)
ENVIRONMENT   CHAR(4)
SOCIALEQUALITY   CHAR(4)
FOUNDED   VARCHAR(25)
FIRSTLOGIN   INTEGER
LASTLOGIN   INTEGER
LASTACTIVITY   VARCHAR(25)
INFLUENCE   VARCHAR(25)

Message Data
Name Data Type

Code: Select all

TIMESTAMP   INTEGER
NATION   VARCHAR(22)
MESSAGE   NVARCHAR(4000)

As far as we can tell, none of the above fields will ever be null (they seem to have a zero when they are empty)

Thank you.

[[violet]]

Here are some guesses we have put together

We don't use a database, so none of the fields are defined in SQL like that. They're all essentially infinite-length strings. That said, your guesses look accurate to me in terms of practical limits.

[NewTexas]

The next thing we would like to know is back to our other post about the digest/catalog(ue)/listing of all the nations and all the regions.

Will that be a possibility? It is not a prob if you don't want to. It just creates a need to create a spider to walk the regions and nations pages of The World to find out tha names of the 56,360 nations and 6,341 regions.

[[violet]]

The next thing we would like to know is back to our other post about the digest/catalog(ue)/listing of all the nations and all the regions.

Will that be a possibility? It is not a prob if you don't want to. It just creates a need to create a spider to walk the regions and nations pages of The World to find out tha names of the 56,360 nations and 6,341 regions.

Yep, will do.

I'm thinking if you want to catalogue the world, you really only need a list of regions, since in loading all of those you also get a list of nations. I can add a nation list easily enough but it will probably be 1-2MB long.

[[violet]]

I've updated the API page with documentation for new 'allregions' and 'allnations' feeds.

Please note that these feeds are strictly rate-limited:

Limit on World streams: 1 request per hour. This limit is low because the data returned is quite large and only changes twice a day (during daily updates): there is no need to query it frequently. Scripts that violate the limit will be permanently banned.

Hope this helps.

[Winnipeg]

This kinda opens the potential for a new "World GDP" script if someone felt inclined to try and put it together... I kinda like the idea.

Just making sure....is there any leaway on the 1 hour thing for those of us developing a script to use it...I find that while in production I can limit the script no problem, when I'm developing it I tend to run a script multiple times to make sure it works...mmm...I suppose I could download the output and save it as an XML File on my local server and just "pretend" I'm pulling it from the server...yeah that's probably a better idea.

Sorry, just hand to ramble a bit to think of my own solution.

[Unibot]

This kinda opens the potential for a new "World GDP" script if someone felt inclined to try and put it together... I kinda like the idea.

Wicked idea, implementing long-term tracking and a separate/combined stats for the World Assembly's member nations would be the icing on the cake.

[NewTexas]

In case you did not know, the nation feed only shows the first 10635 nations...
j/k

The region feeds does have all 6350 regions.
yay

BTW, a date on those feeds would be nice.
maybe as an attribute or something, like:

Code: Select all

<NATIONS date="20100411">

or

Code: Select all

<NATIONS>
<DATE>20100411</DATE>

Also, any idea approximately what time(s) they will be refreshed?

[Winnipeg]

Course, just doing some calculations on the nation counts and the like right now...If I was to limit my requests to say, 1 request every 5 seconds, my collection script would take 8.8 Hours to collect all the region data of the world, and then after that it would take a further 77.7 Hours to do each of the nation's nation data xml for a total of 86.5 Hours, or 3.6 Days per collection cycle.

If we do it 1 request per second, (still meets the 50 per 30 second rate limit) then I could manage 1.76 Hours for the Region Data, and 15.6 Hours for the nations. This would be more acceptable since it would result in less than a 24 hour turn around.

This is of course, not counting time to download the nation's flag...I'm not sure if the rate limit is counting that into account, but for most of my scripts I've been downloading a copy of the nation's flag so that when users access my Census apps they aren't leeching off NationStates' bandwidth to display the flag every time. Of course, if it's a standard flag, then I don't have to worry about the downloading as I already have them, but if its a custom flag, then it's going to have to be downloaded, so that probably adds into the rate limit so again, I'm over the 24 hour mark if I increase the time. Of course, perhaps it doesn't count towards the rate limit since its just a simple HTML request and doesn't involve any real processing on the server's end to generate since it's a static file rather than dynamic.

I don't know if NS currently tracks the age of an uploaded flag, but perhaps an upload date could be added to the API data for future flags so that if a nation has the same flag as the last time it was collected, our scripts could just skip downloading the flag. Since I'm sure alot of nations change their flag once and then never again.

If you're targetting just a single region then the times are much easier to deal with.

[Unibotian WA Mission]

I don't know if NS currently tracks the age of an uploaded flag, but perhaps an upload date could be added to the API data for future flags so that if a nation has the same flag as the last time it was collected, our scripts could just skip downloading the flag. Since I'm sure alot of nations change their flag once and then never again.

If you upload a new flag, its address is different. For example my flag was http://www.nationstates.net/images/flag ... 312475.jpg, but now it is http://www.nationstates.net/images/flag ... 122968.jpg. All I did was upload the same picture.

[Winnipeg]

oh hey, that's new. It used to be just simply "name.jpg" before. Didn't notice they had added that in. That should simplify that.

[Coffee and Crack]

Would it be possible to add the population founder and WA delegate to the api.cgi?allregions=1 script?

[[violet]]

Just making sure....is there any leaway on the 1 hour thing for those of us developing a script to use it...

Yes, if you set your script's UserAgent so I can tell who you are. Currently I'm monitoring this manually, so hitting it a few times within an hour while you're tweaking scripts is fine.

[[violet]]

In case you did not know, the nation feed only shows the first 10635 nations...

Ah. So that's why it's a small download. Okay, I will look into that.

Also, any idea approximately what time(s) they will be refreshed?

At the completion of each daily update, which is around 10:30 am & pm PDT.

[[violet]]

I'm not sure if the rate limit is counting that

The rate limit applies only to API requests.

perhaps an upload date could be added to the API data for future flags so that if a nation has the same flag as the last time it was collected, our scripts could just skip downloading the flag.

As mentioned above, flags have unique filenames these days. If the filename is the same, it's the same image.

[[violet]]

Would it be possible to add the population founder and WA delegate to the api.cgi?allregions=1 script?

Yes, although not trivially. I'd like to understand how you'd use it so I can give you the data you need in the most efficient way.

Here's an alternative I'm now thinking about. It sounds like NSEconomy and NSDossier at least, possibly New Texas and others, need daily info dumps on every region and nation in the game. If that's the case, it may be best for me to make available a single ginormous file, compiled at daily update time, containing every region and nation XML file. That way you'd download one very large file rather than making 60,000 background API requests per day.

If there was a single megaXML file that you could download once per day (only), would that make your life easier and eliminate your need for background scraping jobs? (I'd imagine you'd still send API requests to get up-to-the-second refreshes of data on individual nations and regions, when for example I go to your site and type in my nation name.)

[Coffee and Crack]

I was hope to take the data and provide charts for all the regions like I did for Grand Central

Http://ns-cw.com/mkboard/population.php

[NewTexas]

Here's an alternative I'm now thinking about. It sounds like NSEconomy and NSDossier at least, possibly New Texas and others, need daily info dumps on every region and nation in the game. If that's the case, it may be best for me to make available a single ginormous file, compiled at daily update time, containing every region and nation XML file. That way you'd download one very large file rather than making 60,000 background API requests per day.

Thank you for your offer, but don't do that just for us. We are having enough trouble staying under one a second. NSDossier is going to take a total rewrite. That will take a while.

Maybe we should switch it to a cgi-bin script then all we would have to do is sleep(1).

[Winnipeg]

PHP would server the same purpose... sleep(1) or usleep(1000).

As for the one large file, it could work for sure. It would work really well for something like a GDP calculator I think, since then the data would be all the same age, something like how real nations have their "Quarterly GDP Growth" press releases and stats 2-3 months after the quarter finished.

would be a rather large file but it would probably end up being less bandwidth in the end with only 1 HTTP request/response.

[Pythagosaurus]

It might be somewhat large, but it's all plaintext. We can gzip if it's a problem (or request that scripts include "Accept-Encoding: gzip" in their headers).

[[violet]]

Here's an alternative I'm now thinking about. It sounds like NSEconomy and NSDossier at least, possibly New Texas and others, need daily info dumps on every region and nation in the game. If that's the case, it may be best for me to make available a single ginormous file, compiled at daily update time, containing every region and nation XML file. That way you'd download one very large file rather than making 60,000 background API requests per day.

Thank you for your offer, but don't do that just for us. We are having enough trouble staying under one a second.

A single Whopping Great XML File is better for us than 60,000 API requests. The API requests need to be dynamically generated, and each one has connection overhead.

[Winnipeg]

gzip away Nothing wrong with that, at least for those of us on Apache based servers.

[Coffee and Crack]

I have no issues with gzip either.

[NewTexas]

Upon further reflection, a gzip or zip file is probably a better way to go. It is hard to keep a long running process running for long on a web server. One big download a day would be much easier.