NationStates API (nationdata/regiondata)

[[violet]]

I also think Fris is mistaken, and most if not all scripts currently running do check for this.

I received several double TGs from recruiters when I created a new nation and moved it between feeders.

[[violet]]

I'm not sure if my request was overlooked or in some way impossible.

I'm developing a tool that will run some stats on WA nations in my region. As far as I can tell what I will have to do initially is get the regions nation shard, and then step through each nation to discover if they are WA members, and then assess who is endorsing who.

That's going to create a lot of API calls so it would be extremely helpful if I could get a shard added to the regions feed to just call out the WA nations.

If that's not possible or if I have overlooked it, please let me know.

The way you describe is the only way to do it at the moment, I'm afraid. That is, there is no quick and easy method for getting a list of WA member nations in a region.

Edit: Although your third step is unnecessary. When you loop through the nations, you can request their WA status and endorsements at the same time.

[[violet]]

What info are you gathering? If you want values on a single scale for 2-15 nations, then yeah, I can see that HTML scraping is easier. But the API lets you gather data on lots of scales at once, and at a faster rate, so it's just different implementations of the same thing.

The Influence score. I don't really care about all the other scores of the non-WA nations in TNP, but the Influence score I want to track.

Since The North Pacific has 4,900 nations, isn't it faster to use a Daily Dump? At 15 nations per page, HTML scraping will require 327 requests, which at a rate of 10/minute will take more than half an hour. Even via an API, we don't really have the capacity to load and analyze thousands of nations in a short period of time, which is why we make the Dumps available.

[[violet]]

It just seems counter productive to have to parse all of the nations when you are only interested in a handful.

The thing about the Daily Dumps, though, is they're relatively low-cost for us to generate, because we do it during the daily update, when all the nations need to be loaded and processed anyway. Also, it doesn't matter how many people access them: there's no extra load created. When you make an API call (or scrape an HTML page), though, that's all pure additional load.

[[violet]]

Sorry to be asking a lot of things. Also, not certain where exactly to ask this.

What is the averaging time for API requests? I assume I can't send 50 requests in 1 second and then none for 29 and get away with it? I want to know for the case where I have to perform several API requests that need to be done together. I'd like to do them quickly, but not so quickly that my script gets banned. (and of course limit my overall rate to 50 per 30 seconds)

We only measure in 30-second blocks at the moment, so technically you could make 50 simultaneous requests, then none for 29 seconds, and not fall foul of the auto-banner. However, this would not be a good practice, and may be interpreted as a Denial of Service attack by servers or proxies somewhere along the network chain (including our provider).

Modern browsers limit themselves to between 2 and 6 simultaneous connections, but this is a pretty high number, balanced out by the fact that browsers tend to sit idle afterward while the user reads the web page.

Unless you have a very good reason, I'd say that a single-purpose script should only make 1 simultaneous request. That is, it should request data and wait until it receives it before asking for more. (You get a response in under half a second anyway, so your limiting factor is still the Rate Limit.) This is decidedly so if your script runs in response to user input; e.g. it is a web page that fetches data whenever a visitor clicks something. In this case, when five visitors come along to your site at once, you probably generate five times the usual number of simultaneous API connections.

Also, you may have (or develop) a couple of different kinds of scripts, which run in parallel. So even if you keep your scripts to 1 simultaneous connection each, you may wind up making quite a lot in total.

[Cerberion]

It just seems counter productive to have to parse all of the nations when you are only interested in a handful.

The thing about the Daily Dumps, though, is they're relatively low-cost for us to generate, because we do it during the daily update, when all the nations need to be loaded and processed anyway. Also, it doesn't matter how many people access them: there's no extra load created. When you make an API call (or scrape an HTML page), though, that's all pure additional load.

Thanks for the feedback. It's greatly appreciated.

[Zemnaya Svoboda]

The Influence score. I don't really care about all the other scores of the non-WA nations in TNP, but the Influence score I want to track.

Since The North Pacific has 4,900 nations, isn't it faster to use a Daily Dump? At 15 nations per page, HTML scraping will require 327 requests, which at a rate of 10/minute will take more than half an hour. Even via an API, we don't really have the capacity to load and analyze thousands of nations in a short period of time, which is why we make the Dumps available.

I mean influence score not influence rank. The various scores are not saved in the daily data dump. If they were, it would become rather large.

Sorry to be asking a lot of things. Also, not certain where exactly to ask this.

What is the averaging time for API requests? I assume I can't send 50 requests in 1 second and then none for 29 and get away with it? I want to know for the case where I have to perform several API requests that need to be done together. I'd like to do them quickly, but not so quickly that my script gets banned. (and of course limit my overall rate to 50 per 30 seconds)

We only measure in 30-second blocks at the moment, so technically you could make 50 simultaneous requests, then none for 29 seconds, and not fall foul of the auto-banner. However, this would not be a good practice, and may be interpreted as a Denial of Service attack by servers or proxies somewhere along the network chain (including our provider).

Modern browsers limit themselves to between 2 and 6 simultaneous connections, but this is a pretty high number, balanced out by the fact that browsers tend to sit idle afterward while the user reads the web page.

Unless you have a very good reason, I'd say that a single-purpose script should only make 1 simultaneous request. That is, it should request data and wait until it receives it before asking for more. (You get a response in under half a second anyway, so your limiting factor is still the Rate Limit.) This is decidedly so if your script runs in response to user input; e.g. it is a web page that fetches data whenever a visitor clicks something. In this case, when five visitors come along to your site at once, you probably generate five times the usual number of simultaneous API connections.

Also, you may have (or develop) a couple of different kinds of scripts, which run in parallel. So even if you keep your scripts to 1 simultaneous connection each, you may wind up making quite a lot in total.

I've long been using a single process which receives requests from (nearly) all of my scripts and then actually fetches them from nationstates.net, with 601 ms gaps between requests. I've long intended to clean up the code a bit and publish it. Perhaps sometime soon I will.

It's a model which seems to work well for me.

[Cerberion]

Mildly off topic but I did decide to parse the nations.xml feed with nokogiri.

It parsed out the xml in 9.5 seconds, including the time to download the nations.xml feed from NS.

Just thought I'd share that as I was impressed greatly by the performance and, as stated previously, attempting to parse the XML in ruby using SAX2 was dreadful.

I'll stop my whining now and be satisfied with the nations.xml.
Thanks again for the folks that tried to guide me in the right direction. It's very much appreciated.

[Zemnaya Svoboda]

Mildly off topic but I did decide to parse the nations.xml feed with nokogiri.

It parsed out the xml in 9.5 seconds, including the time to download the nations.xml feed from NS.

Just thought I'd share that as I was impressed greatly by the performance and, as stated previously, attempting to parse the XML in ruby using SAX2 was dreadful.

I'll stop my whining now and be satisfied with the nations.xml.
Thanks again for the folks that tried to guide me in the right direction. It's very much appreciated.

I believe nokogiri uses libxml2, a pretty fast C library.

[Poopcannon]

Sorry to be asking a lot of things. Also, not certain where exactly to ask this.

What is the averaging time for API requests? I assume I can't send 50 requests in 1 second and then none for 29 and get away with it? I want to know for the case where I have to perform several API requests that need to be done together. I'd like to do them quickly, but not so quickly that my script gets banned. (and of course limit my overall rate to 50 per 30 seconds)

We only measure in 30-second blocks at the moment, so technically you could make 50 simultaneous requests, then none for 29 seconds, and not fall foul of the auto-banner. However, this would not be a good practice, and may be interpreted as a Denial of Service attack by servers or proxies somewhere along the network chain (including our provider).

Modern browsers limit themselves to between 2 and 6 simultaneous connections, but this is a pretty high number, balanced out by the fact that browsers tend to sit idle afterward while the user reads the web page.

Unless you have a very good reason, I'd say that a single-purpose script should only make 1 simultaneous request. That is, it should request data and wait until it receives it before asking for more. (You get a response in under half a second anyway, so your limiting factor is still the Rate Limit.) This is decidedly so if your script runs in response to user input; e.g. it is a web page that fetches data whenever a visitor clicks something. In this case, when five visitors come along to your site at once, you probably generate five times the usual number of simultaneous API connections.

Also, you may have (or develop) a couple of different kinds of scripts, which run in parallel. So even if you keep your scripts to 1 simultaneous connection each, you may wind up making quite a lot in total.

Thanks for the info. I've always kept these ideas in mind and cache as much as possible. The reason I'm asking is for processes where it is statistically beneficial not to spread my requests out evenly.

[Johz]

I was just wondering if any of the API shard/tags return a value of 'None' in any situation?

[Fischistan]

I was just wondering if any of the API shard/tags return a value of 'None' in any situation?

A few of the shards return 0, like< FIRSTLOGIN> if they are antiquity, and delegate/founder of the region api if they don't have one.

[Johz]

I was just wondering if any of the API shard/tags return a value of 'None' in any situation?

A few of the shards return 0, like< FIRSTLOGIN> if they are antiquity, and delegate/founder of the region api if they don't have one.

Thanks.

[ALUCARD GRIMUS]

I'm not sure if this issue was risen by anyone yet, but the daily data dump (this one: http://www.nationstates.net/pages/regions.xml.gz) has several regions that repeat themselves.

For example:

Goauld System Lords
Ether
Ether
Ether
Ether
Ether
Ether
Kingdom of Northern Ireland
Novels and Plays
Novels and Plays
Ionia
Karrhai
Amarr Empire
The Free Land
Magicka
Chukaku
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
USSR and Eastern Europe
USSR and Eastern Europe
USSR and Eastern Europe
USSR and Eastern Europe
USSR and Eastern Europe

Is this supposed to happen?

[Spartzerium]

I'm not sure if this issue was risen by anyone yet, but the daily data dump (this one: http://www.nationstates.net/pages/regions.xml.gz) has several regions that repeat themselves.

For example:

Goauld System Lords
Ether
Ether
Ether
Ether
Ether
Ether
Kingdom of Northern Ireland
Novels and Plays
Novels and Plays
Ionia
Karrhai
Amarr Empire
The Free Land
Magicka
Chukaku
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
USSR and Eastern Europe
USSR and Eastern Europe
USSR and Eastern Europe
USSR and Eastern Europe
USSR and Eastern Europe

Is this supposed to happen?

Doesn't seem to be a problem with the data dump itself, works just fine for me. I'd assume it's an issue with the program you're opening it with.

[ALUCARD GRIMUS]

I'm not sure if this issue was risen by anyone yet, but the daily data dump (this one: http://www.nationstates.net/pages/regions.xml.gz) has several regions that repeat themselves.

For example:

Goauld System Lords
Ether
Ether
Ether
Ether
Ether
Ether
Kingdom of Northern Ireland
Novels and Plays
Novels and Plays
Ionia
Karrhai
Amarr Empire
The Free Land
Magicka
Chukaku
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
The Isle of Wight
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
Galapagos
USSR and Eastern Europe
USSR and Eastern Europe
USSR and Eastern Europe
USSR and Eastern Europe
USSR and Eastern Europe

Is this supposed to happen?

Doesn't seem to be a problem with the data dump itself, works just fine for me. I'd assume it's an issue with the program you're opening it with.

I've opened it with Notepad++ and Excel 2010, and it happened in both, so I find that unlikely.

[Spartzerium]

Doesn't seem to be a problem with the data dump itself, works just fine for me. I'd assume it's an issue with the program you're opening it with.

I've opened it with Notepad++ and Excel 2010, and it happened in both, so I find that unlikely.

I am able to reproduce the problem with Excel 2007 (don't have 2010). Seems to be caused by the embassies. The XML source appears perfectly fine to me though.

[Ballotonia]

If you don't have an xsd file (which NS doesn't provide) which tells Excel what the structure of the XML file is, Excel will try to make up something by itself. It's really bad at doing so. As a consequence, in the Excel table the rows are duplicated, one duplication for each next embassy listed.

Possible solutions are to remove the embassy information before opening in Excel, supplying Excel with an (own made) xsd file, or using a different tool.

And as an avid Notepad++ user I can say for sure that this doesn't happen when opening the data dump file directly in Notepad++. If you however save Excel to .txt, then open in Notepad++ (or any other text editor), then yes... but that's because Excel has already ruined the data.

Ballotonia

[Scoochi2]

Would it be possible to add in an extra shard to indicate whether a region has a password or not?
Could this be included in the daily dump?

Possible examples:

Code: Select all

<PASSWORDED>None</PASSWORDED>

<PASSWORDED>Displayed</PASSWORDED>

<PASSWORDED>Hidden</PASSWORDED>

[Unibot II]

Would it be possible to add in an extra shard to indicate whether a region has a password or not?
Could this be included in the daily dump?

Possible examples:

Code: Select all

<PASSWORDED>None</PASSWORDED>

<PASSWORDED>Displayed</PASSWORDED>

<PASSWORDED>Hidden</PASSWORDED>

This would be very helpful.

[SunRawr]

Not hard at all!

http://www.nationstates.net/cgi-bin/api ... lms&q=tags

Is it also possible to have it the other way around? Ie. An API that shows which regions have @@TAG@@ ?

Either way, I see now a possibility to do some multiple tag searches on a daily db dump(it is/will be included in the dump right?) to do some multiple TAG searches (eg. combining Democratic and Founderless). *goes off to sleep and sees some nice coding ahead*

I couldn't find an answer to this and wanted to bring it back up. It would be very useful.

[Fischistan]

Do the national events RSS feeds count as the API? Because I would like to know if we could access those at a rate similar to the API rate.

[Ballotonia]

Is it also possible to have it the other way around? Ie. An API that shows which regions have @@TAG@@ ?

Either way, I see now a possibility to do some multiple tag searches on a daily db dump(it is/will be included in the dump right?) to do some multiple TAG searches (eg. combining Democratic and Founderless). *goes off to sleep and sees some nice coding ahead*

I couldn't find an answer to this and wanted to bring it back up. It would be very useful.

Yes, not only do I agree this would be a useful thing to add to the API, I already coded it. I'm just waiting for final approval for this one to go live.

Ballotonia

[The BlAAtschApen]

I couldn't find an answer to this and wanted to bring it back up. It would be very useful.

Yes, not only do I agree this would be a useful thing to add to the API, I already coded it. I'm just waiting for final approval for this one to go live.

Ballotonia

Woooo!

Thanks in advance. This saves me a lot of work. I had already designed a (clunky) way to do multiple tag searches on a dump(not what it sounds like, people). Now it can go much easier and I can have live searches.

[SunRawr]

I couldn't find an answer to this and wanted to bring it back up. It would be very useful.

Yes, not only do I agree this would be a useful thing to add to the API, I already coded it. I'm just waiting for final approval for this one to go live.

Ballotonia

Yay!