NATION

PASSWORD

NationStates API (nationdata/regiondata)

Bug reports, general help, ideas for improvements, and questions about how things are meant to work.
User avatar
[violet]
Site Admin
 
Posts: 14065
Founded: Antiquity

NationStates API (nationdata/regiondata)

Postby [violet] » Fri Apr 09, 2010 12:14 am

I've made a few changes to the NS API. By which I mean this:

http://www.nationstates.net/pages/api.html

This has previously been known as regiondata.cgi and nationdata.cgi, both of which continue to work at their old addresses. They essentially forward to this one now. which have been retired as of August 2013.

The API exists so that people can build NS-based sites, such as NSDossier and NSEconomy. We want to support such efforts.

Please visit the API documentation page for information. The major change is that very high rates of queries (more than 100/minute) will result in the IP address being denied service for 15 minutes. This is to ensure regular players don't suffer slowdowns because of overzealous scripts.

In general, scripts should not attempt to look up every resident of a region all at once. This can result in dozens, hundreds, or thousands of near-simultaneous requests to the server, which are difficult or impossible to service. If you need information on large numbers of nations, you should trawl them from the API as a speed-limited background job and store the data locally on your server.

Please let me know if you have any questions or concerns!

Background: This is the result of events documented in the "Scripts banned" thread.

Latest Change: 5-Oct-2014. API version bumped from 6 to 7. The standard Nation API now includes <FACTBOOKS> and <DISPATCHES>, which display the number of each created by that nation. These are also new Nation API shards ("factbooks", "dispatches"), as are "factbooklist" and "dispatchlist", which displays a list of documents created by this nation.
Last edited by [violet] on Sat Oct 04, 2014 10:27 pm, edited 5 times in total.
Reason: Retired nationdata.cgi & regiondata.cgi

User avatar
NewTexas
Spokesperson
 
Posts: 168
Founded: Antiquity
Anarchy

Postby NewTexas » Fri Apr 09, 2010 10:18 am

That is great. A very important characteristic of an API is details. Now that you are encouraging developers to use a database to store the data versus just hitting the feeds any time they want it, some details are needed. In particular, information is needed about the data such as type and length. When getting data from the feeds, it did not matter because it was never being stored anywhere. When creating a database, you need to know how what kind of things you are going to put in it and how big they are possibly going to be.

Here are some guesses we have put together, if you could confirm and/or correct the following information, that would be great:

Region Data
Name Data Type
Code: Select all
NAME   VARCHAR(30)
FACTBOOK   NVARCHAR(800)
NUMNATIONS   INTEGER
NATIONS   BLOB
DELEGATE   VARCHAR(22)
DELEGATEVOTES   INTEGER
FOUNDER   VARCHAR(22)
POWER   VARCHAR(30)
FLAG   VARCHAR(50)


Nation Data
Name Data Type
Code: Select all
NAME   VARCHAR(22)
TYPE   VARCHAR(40)
FULLNAME   VARCHAR(63)
MOTTO   NVARCHAR(125)
CATEGORY   VARCHAR(25)
UNSTATUS   CHAR(2)
CIVILRIGHTS   VARCHAR(25)
ECONOMY   VARCHAR(25)
POLITICALFREEDOM   VARCHAR(25)
REGION   VARCHAR(30)
POPULATION   INTEGER
TAX   INTEGER
ANIMAL   VARCHAR(25)
CURRENCY   VARCHAR(25)
FLAG   VARCHAR(50)
MAJORINDUSTRY   VARCHAR(50)
GOVTPRIORITY   VARCHAR(50)
ADMINISTRATION   CHAR(4)
WELFARE   CHAR(4)
HEALTHCARE   CHAR(4)
EDUCATION   CHAR(4)
SPIRITUALITY   CHAR(4)
DEFENCE   CHAR(4)
LAWANDORDER   CHAR(4)
COMMERCE   CHAR(4)
PUBLICTRANSPORT   CHAR(4)
ENVIRONMENT   CHAR(4)
SOCIALEQUALITY   CHAR(4)
FOUNDED   VARCHAR(25)
FIRSTLOGIN   INTEGER
LASTLOGIN   INTEGER
LASTACTIVITY   VARCHAR(25)
INFLUENCE   VARCHAR(25)



Message Data
Name Data Type
Code: Select all
TIMESTAMP   INTEGER
NATION   VARCHAR(22)
MESSAGE   NVARCHAR(4000)


As far as we can tell, none of the above fields will ever be null (they seem to have a zero when they are empty)

Thank you.
Last edited by NewTexas on Fri Apr 09, 2010 10:24 am, edited 1 time in total.
Big Tex
Governor of Texas

Author: NSDossier

User avatar
[violet]
Site Admin
 
Posts: 14065
Founded: Antiquity

Postby [violet] » Fri Apr 09, 2010 4:01 pm

NewTexas wrote:Here are some guesses we have put together

We don't use a database, so none of the fields are defined in SQL like that. They're all essentially infinite-length strings. That said, your guesses look accurate to me in terms of practical limits.

User avatar
NewTexas
Spokesperson
 
Posts: 168
Founded: Antiquity
Anarchy

Postby NewTexas » Fri Apr 09, 2010 5:21 pm

The next thing we would like to know is back to our other post about the digest/catalog(ue)/listing of all the nations and all the regions.

Will that be a possibility? It is not a prob if you don't want to. It just creates a need to create a spider to walk the regions and nations pages of The World to find out tha names of the 56,360 nations and 6,341 regions.

:bow:
Big Tex
Governor of Texas

Author: NSDossier

User avatar
[violet]
Site Admin
 
Posts: 14065
Founded: Antiquity

Postby [violet] » Fri Apr 09, 2010 6:27 pm

NewTexas wrote:The next thing we would like to know is back to our other post about the digest/catalog(ue)/listing of all the nations and all the regions.

Will that be a possibility? It is not a prob if you don't want to. It just creates a need to create a spider to walk the regions and nations pages of The World to find out tha names of the 56,360 nations and 6,341 regions.

Yep, will do.

I'm thinking if you want to catalogue the world, you really only need a list of regions, since in loading all of those you also get a list of nations. I can add a nation list easily enough but it will probably be 1-2MB long.

User avatar
[violet]
Site Admin
 
Posts: 14065
Founded: Antiquity

Postby [violet] » Fri Apr 09, 2010 7:24 pm

I've updated the API page with documentation for new 'allregions' and 'allnations' feeds.

Please note that these feeds are strictly rate-limited:
Limit on World streams: 1 request per hour. This limit is low because the data returned is quite large and only changes twice a day (during daily updates): there is no need to query it frequently. Scripts that violate the limit will be permanently banned.

Hope this helps.

User avatar
Winnipeg
Bureaucrat
 
Posts: 52
Founded: Antiquity
Benevolent Dictatorship

Postby Winnipeg » Sat Apr 10, 2010 8:46 am

This kinda opens the potential for a new "World GDP" script if someone felt inclined to try and put it together... I kinda like the idea.

Just making sure....is there any leaway on the 1 hour thing for those of us developing a script to use it...I find that while in production I can limit the script no problem, when I'm developing it I tend to run a script multiple times to make sure it works...mmm...I suppose I could download the output and save it as an XML File on my local server and just "pretend" I'm pulling it from the server...yeah that's probably a better idea.

Sorry, just hand to ramble a bit to think of my own solution.
Robin 'Winnipeg' Morrison
Creator of WinniVision(TM)
Home of the West Pacific

User avatar
Unibot
Senator
 
Posts: 4292
Founded: May 25, 2008
Ex-Nation

Postby Unibot » Sat Apr 10, 2010 1:18 pm

Winnipeg wrote:This kinda opens the potential for a new "World GDP" script if someone felt inclined to try and put it together... I kinda like the idea.


Wicked idea, implementing long-term tracking and a separate/combined stats for the World Assembly's member nations would be the icing on the cake.

User avatar
NewTexas
Spokesperson
 
Posts: 168
Founded: Antiquity
Anarchy

Postby NewTexas » Sun Apr 11, 2010 9:56 am

In case you did not know, the nation feed only shows the first 10635 nations...
:palm: j/k

The region feeds does have all 6350 regions.
:clap: yay

BTW, a date on those feeds would be nice.
maybe as an attribute or something, like:
Code: Select all
<NATIONS date="20100411">

or
Code: Select all
<NATIONS>
<DATE>20100411</DATE>


:geek:

Also, any idea approximately what time(s) they will be refreshed?

:?:
Big Tex
Governor of Texas

Author: NSDossier

User avatar
Winnipeg
Bureaucrat
 
Posts: 52
Founded: Antiquity
Benevolent Dictatorship

Postby Winnipeg » Sun Apr 11, 2010 11:51 am

Course, just doing some calculations on the nation counts and the like right now...If I was to limit my requests to say, 1 request every 5 seconds, my collection script would take 8.8 Hours to collect all the region data of the world, and then after that it would take a further 77.7 Hours to do each of the nation's nation data xml for a total of 86.5 Hours, or 3.6 Days per collection cycle.

If we do it 1 request per second, (still meets the 50 per 30 second rate limit) then I could manage 1.76 Hours for the Region Data, and 15.6 Hours for the nations. This would be more acceptable since it would result in less than a 24 hour turn around.

This is of course, not counting time to download the nation's flag...I'm not sure if the rate limit is counting that into account, but for most of my scripts I've been downloading a copy of the nation's flag so that when users access my Census apps they aren't leeching off NationStates' bandwidth to display the flag every time. Of course, if it's a standard flag, then I don't have to worry about the downloading as I already have them, but if its a custom flag, then it's going to have to be downloaded, so that probably adds into the rate limit so again, I'm over the 24 hour mark if I increase the time. Of course, perhaps it doesn't count towards the rate limit since its just a simple HTML request and doesn't involve any real processing on the server's end to generate since it's a static file rather than dynamic.

I don't know if NS currently tracks the age of an uploaded flag, but perhaps an upload date could be added to the API data for future flags so that if a nation has the same flag as the last time it was collected, our scripts could just skip downloading the flag. Since I'm sure alot of nations change their flag once and then never again.

If you're targetting just a single region then the times are much easier to deal with.
Robin 'Winnipeg' Morrison
Creator of WinniVision(TM)
Home of the West Pacific

User avatar
Unibotian WA Mission
Chargé d'Affaires
 
Posts: 432
Founded: Oct 20, 2009
Ex-Nation

Postby Unibotian WA Mission » Sun Apr 11, 2010 12:56 pm

Winnipeg wrote:I don't know if NS currently tracks the age of an uploaded flag, but perhaps an upload date could be added to the API data for future flags so that if a nation has the same flag as the last time it was collected, our scripts could just skip downloading the flag. Since I'm sure alot of nations change their flag once and then never again.


If you upload a new flag, its address is different. For example my flag was http://www.nationstates.net/images/flag ... 312475.jpg, but now it is http://www.nationstates.net/images/flag ... 122968.jpg. All I did was upload the same picture.
Vocenae wrote:Unibot, you have won NS.
General Halcones wrote: Look up to Unibot as an example.
Member of Gholgoth | The Capitalis de Societate of The United Defenders League (UDL) | Org. Join Date: 25/05/2008
Unibotian Factbook // An Analysis of NationStates Generations // The Gameplay Alignment Test // NS Weather // How do I join the UDL?
World Assembly Card Gallery // The Unibotian Life Expectancy Index // Proudly Authored 9 GA Res., 14 SC Res. // Commended by SC#78;

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
✯ Duty is Eternal, Justice is Imminent: UDL

User avatar
Winnipeg
Bureaucrat
 
Posts: 52
Founded: Antiquity
Benevolent Dictatorship

Postby Winnipeg » Sun Apr 11, 2010 1:13 pm

oh hey, that's new. It used to be just simply "name.jpg" before. Didn't notice they had added that in. That should simplify that.
Robin 'Winnipeg' Morrison
Creator of WinniVision(TM)
Home of the West Pacific

User avatar
Coffee and Crack
Bureaucrat
 
Posts: 59
Founded: Aug 17, 2008
Ex-Nation

Postby Coffee and Crack » Sun Apr 11, 2010 2:34 pm

Would it be possible to add the population founder and WA delegate to the api.cgi?allregions=1 script?

User avatar
[violet]
Site Admin
 
Posts: 14065
Founded: Antiquity

Postby [violet] » Sun Apr 11, 2010 10:28 pm

Winnipeg wrote:Just making sure....is there any leaway on the 1 hour thing for those of us developing a script to use it...

Yes, if you set your script's UserAgent so I can tell who you are. Currently I'm monitoring this manually, so hitting it a few times within an hour while you're tweaking scripts is fine.

User avatar
[violet]
Site Admin
 
Posts: 14065
Founded: Antiquity

Postby [violet] » Sun Apr 11, 2010 10:34 pm

NewTexas wrote:In case you did not know, the nation feed only shows the first 10635 nations...

Ah. So that's why it's a small download. Okay, I will look into that.

Also, any idea approximately what time(s) they will be refreshed?

At the completion of each daily update, which is around 10:30 am & pm PDT.

User avatar
[violet]
Site Admin
 
Posts: 14065
Founded: Antiquity

Postby [violet] » Sun Apr 11, 2010 10:37 pm

Winnipeg wrote:I'm not sure if the rate limit is counting that

The rate limit applies only to API requests.

perhaps an upload date could be added to the API data for future flags so that if a nation has the same flag as the last time it was collected, our scripts could just skip downloading the flag.

As mentioned above, flags have unique filenames these days. If the filename is the same, it's the same image.

User avatar
[violet]
Site Admin
 
Posts: 14065
Founded: Antiquity

Postby [violet] » Sun Apr 11, 2010 10:49 pm

Coffee and Crack wrote:Would it be possible to add the population founder and WA delegate to the api.cgi?allregions=1 script?

Yes, although not trivially. I'd like to understand how you'd use it so I can give you the data you need in the most efficient way.

Here's an alternative I'm now thinking about. It sounds like NSEconomy and NSDossier at least, possibly New Texas and others, need daily info dumps on every region and nation in the game. If that's the case, it may be best for me to make available a single ginormous file, compiled at daily update time, containing every region and nation XML file. That way you'd download one very large file rather than making 60,000 background API requests per day.

If there was a single megaXML file that you could download once per day (only), would that make your life easier and eliminate your need for background scraping jobs? (I'd imagine you'd still send API requests to get up-to-the-second refreshes of data on individual nations and regions, when for example I go to your site and type in my nation name.)

User avatar
Coffee and Crack
Bureaucrat
 
Posts: 59
Founded: Aug 17, 2008
Ex-Nation

Postby Coffee and Crack » Mon Apr 12, 2010 5:35 am

I was hope to take the data and provide charts for all the regions like I did for Grand Central

Http://ns-cw.com/mkboard/population.php

User avatar
NewTexas
Spokesperson
 
Posts: 168
Founded: Antiquity
Anarchy

Postby NewTexas » Mon Apr 12, 2010 5:55 pm

[violet] wrote:Here's an alternative I'm now thinking about. It sounds like NSEconomy and NSDossier at least, possibly New Texas and others, need daily info dumps on every region and nation in the game. If that's the case, it may be best for me to make available a single ginormous file, compiled at daily update time, containing every region and nation XML file. That way you'd download one very large file rather than making 60,000 background API requests per day.


Thank you for your offer, but don't do that just for us. We are having enough trouble staying under one a second. NSDossier is going to take a total rewrite. That will take a while.

Maybe we should switch it to a cgi-bin script then all we would have to do is sleep(1).

:(
Big Tex
Governor of Texas

Author: NSDossier

User avatar
Winnipeg
Bureaucrat
 
Posts: 52
Founded: Antiquity
Benevolent Dictatorship

Postby Winnipeg » Tue Apr 13, 2010 6:18 am

PHP would server the same purpose... sleep(1) or usleep(1000).

As for the one large file, it could work for sure. It would work really well for something like a GDP calculator I think, since then the data would be all the same age, something like how real nations have their "Quarterly GDP Growth" press releases and stats 2-3 months after the quarter finished.

would be a rather large file but it would probably end up being less bandwidth in the end with only 1 HTTP request/response.
Robin 'Winnipeg' Morrison
Creator of WinniVision(TM)
Home of the West Pacific

User avatar
Pythagosaurus
Cute Purple Dinosaur
 
Posts: 549
Founded: Nov 24, 2004
Civil Rights Lovefest

Postby Pythagosaurus » Tue Apr 13, 2010 9:59 pm

It might be somewhat large, but it's all plaintext. We can gzip if it's a problem (or request that scripts include "Accept-Encoding: gzip" in their headers).

User avatar
[violet]
Site Admin
 
Posts: 14065
Founded: Antiquity

Postby [violet] » Tue Apr 13, 2010 10:19 pm

NewTexas wrote:
[violet] wrote:Here's an alternative I'm now thinking about. It sounds like NSEconomy and NSDossier at least, possibly New Texas and others, need daily info dumps on every region and nation in the game. If that's the case, it may be best for me to make available a single ginormous file, compiled at daily update time, containing every region and nation XML file. That way you'd download one very large file rather than making 60,000 background API requests per day.


Thank you for your offer, but don't do that just for us. We are having enough trouble staying under one a second.

A single Whopping Great XML File is better for us than 60,000 API requests. The API requests need to be dynamically generated, and each one has connection overhead.

User avatar
Winnipeg
Bureaucrat
 
Posts: 52
Founded: Antiquity
Benevolent Dictatorship

Postby Winnipeg » Wed Apr 14, 2010 6:18 am

gzip away :) Nothing wrong with that, at least for those of us on Apache based servers.
Robin 'Winnipeg' Morrison
Creator of WinniVision(TM)
Home of the West Pacific

User avatar
Coffee and Crack
Bureaucrat
 
Posts: 59
Founded: Aug 17, 2008
Ex-Nation

Postby Coffee and Crack » Wed Apr 14, 2010 10:23 am

I have no issues with gzip either.

User avatar
NewTexas
Spokesperson
 
Posts: 168
Founded: Antiquity
Anarchy

Postby NewTexas » Thu Apr 15, 2010 4:33 pm

Upon further reflection, a gzip or zip file is probably a better way to go. It is hard to keep a long running process running for long on a web server. One big download a day would be much easier.

:)
Big Tex
Governor of Texas

Author: NSDossier

Next

Advertisement

Remove ads

Return to Technical

Who is online

Users browsing this forum: Perlica

Advertisement

Remove ads