NationStates API (nationdata/regiondata)

[Coffee and Crack]

I just use a cronjob every hour for my population tracking script.

[NewTexas]

I just use a cronjob every hour for my population tracking script.

That is not the same as starting a job that will make an request every second for 56,000 seconds in a row, sleep 30,000 seconds in a row, rinse and repeat, 365 times a year. That is the alternative without the one file if you want the whole world.

[[violet]]

I might make two XML files, one containing all regions and the other containing all nations -- just in case anyone wants one but not the other. Sound OK?

[Winnipeg]

Would work for me as long as we can request each one once every 24 hours in case we want both.

[[violet]]

Yep, of course.

I'll look at getting this in this week.

[Winnipeg]

Cool beans, looking forward to it!

[Nanocyberia]

Hello,

I'd like to see a population tracker right within NS, instead of the API. Anywhere from the last 3 days to the last 7... (And then I could make my own tracker where I just copy-and-paste the values into a static HTML file, rather than doing it everyday.) Nothing fancy, either, so it doesn't need to be a graph or anything.
I'd personally also fancy seeing the detailed gov stats somewhere in NS, too, but just single, daily updates; no tracking.

[Winnipeg]

I doubt you'll see that happen anytime soon, and it almost seems kinda pointless for the population itself since it's nearly a static movement every day. I mean you'd have a line graph that went up by 5-8 Million every day it would look pretty stable at a very low incline I'm sure.

Plus NS doesn't use a Database for storing data, so holding this data would probably end up being much harder on the server than it would be for a Database setup.

[Nanocyberia]

I mean you'd have a line graph that went up by 5-8 Million every day it would look pretty stable at a very low incline I'm sure.

Graphs are way too hard... A table would be sufficient; it needs to be dynamically generated, so that only that last four values are kept, for example. Conceptually, it would be like this :

Code: Select all

dump pop. count to table;
if row count >= 5, delete top/bottom row; // delete top row if most recent value is in bottom row; else, other way round :)


I'm pretty sure you don't a database for that... (In fact, if one is feeling really adventurous, you could set up the above script to allow custom values for the conditional statement; let's say the default is no more than three rows. After that, take your pick between a drop-down menu with pre-set choices or an input field with a maximum value.)

[Fit battion]

A graph for regional population might be nice (if necessary I'll make a seperate thread), where it automatically changes at update and calculates pop and stuff.

[North Mack]

Oh look, a straight line...

All the information you want to see is available outside of NS through various calculators, and not really necessary to make the admins and techies put so much work into something that already exists.

[Coffee and Crack]

Any update on the super lists?

[[violet]]

Got delayed by major server issues this week, hoping to get to it soon.

[[violet]]

OK, around 02-May-10 22:20 PDT two files should appear:

http://nationstates.net/pages/nations.xml.gz
http://nationstates.net/pages/regions.xml.gz

... and thereafter be updated once every 24 hours. These will contain the XML of all regions and nations in the game. Please let me know if you find these useful, any feedback regarding data format, etc.

One difference to the regular API is that each item is <NATION> (or <REGION>), not <NATIONINFO> (or <REGIONINFO>).

[NewTexas]

Unable to download either file. "website is unavailable or not found"

[North Mack]

OK, around 02-May-10 22:20 PDT two files should appear:

She said they aren't there yet...

[Winnipeg]

OK, around 02-May-10 22:20 PDT two files should appear:

http://nationstates.net/pages/nations.xml.gz
http://nationstates.net/pages/regions.xml.gz

... and thereafter be updated once every 24 hours. These will contain the XML of all regions and nations in the game. Please let me know if you find these useful, any feedback regarding data format, etc.

One difference to the regular API is that each item is <NATION> (or <REGION>), not <NATIONINFO> (or <REGIONINFO>).

Thanks Violet, a fitting release date for the API update I think My 29th Birthday.

[NewTexas]

Please let me know if you find these useful, any feedback regarding data format, etc.

We would like to let you know, but three different tools were unable to gunzip them into anything meaningful.

Anybody else have any luck?

[Pythagosaurus]

`file` is your friend. It's gzipped twice. 8)

[New South Hell]

I was able to process the regions file with WinZip once I renamed it to .tgz. This is a trick I discovered some time ago that often helps for .gz files downloaded from the Web.

- nsh -

[New South Hell]

I have to correct my previous post. When I downloaded the nations file, I found that what I needed to do was unzip it, rename the extracted file to have a .tgz extension, and then unzip it a second time. I suspect this is what I did with the regions file as well - but that I reported it incorrectly because I had a bit of confusion between the original .gz file and its contents which were still very clearly compressed.

[NewTexas]

Thank you very much for the tip that it was gzipped twice. We never would have guessed that. And we don't even want to know why it is gzipped twice. We have it working and loaded in a database now and a mechanism for doing it daily. The only slight gotcha is that we are going to have to do a purge periodically to get rid of all the dead soldiers.

Now our tool will only hit the actual feeds twice for each request. One to get the latest nation feed for the request and one to get the latest region feed for the request. All the rest of the processing will be done out of the database.

The load times were much better than using the feeds.

Regions
start: 2010-05-04T10:53:34.347
end: 2010-05-04T10:54:03.763

Nations
start: 2010-05-04T15:11:09.120
end: 2010-05-04T15:29:04.533

So, it took
30 seconds to load 6,300+ regions
and
18 minutes to load 52,000+ nations

Much better than the 1 per second, which would have been:
6300 seconds to load 6,300+ regions
and
867 minutes to load 52,000+ nations


WooHootie!

Way to go [violet]!



<now, to recode the rest of our app to use the db instead of the feeds>
<sigh>

[Pythagosaurus]

I wouldn't count on it staying gzipped twice. It was almost certainly a mistake. Somebody probably just forgot to test his/her code.

[[violet]]

Hmm, odd. The file is not double-zipped, but it winds up that way on your computer when you download it via HTTP.

[[violet]]

Possibly fixed for update in 22 hours from now, but possibly not because I'm not really sure what's going on.