NationStates API (nationdata/regiondata)

[Topid]

Boo! I feel like there's quite a few more of us that would like to use endorsements than a break down on cause of death percentages and such!

Is there a way we could get a version that was bigger that would include more shards?

[Ever-Wandering Souls]

Boo! I feel like there's quite a few more of us that would like to use endorsements than a break down on cause of death percentages and such!

Is there a way we could get a version that was bigger that would include more shards?

https://www.nationstates.net/cgi-bin/ap ... dorsements

https://www.nationstates.net/cgi-bin/ap ... dorsements

etc. Unless you want who specifically is endorsing like, everyone in the game, on a basis more up to date than something that polls all WA members for that data. Which, with about 27k members, at 100 requests a minute, would take about four and a half hours to poll the entirety of, if you really needed it. If you wanted to do all WA members in a region, you could pull all nations in a region, cross it with all WA members, and run the above only on WA members in x region, taking significantly less time.

[Topid]

Yeah I don't know how to do all that. I know how to use Excel really really well, but none of that fancy coding to make API requests.

That's why I like the data dump because I can import it to Excel and use filters to show only what I need.

[Ever-Wandering Souls]

Yeah I don't know how to do all that. I know how to use Excel really really well, but none of that fancy coding to make API requests.

That's why I like the data dump because I can import it to Excel and use filters to show only what I need.

Fair, I'm not great with much of it either, but you can manually use shards like that for spot data. And if you want to do a bit more, getting an excel sheet to populate with data from shards is on the more basic end of stuff to learn.

[Topid]

Yeah I don't know how to do all that. I know how to use Excel really really well, but none of that fancy coding to make API requests.

That's why I like the data dump because I can import it to Excel and use filters to show only what I need.

Fair, I'm not great with much of it either, but you can manually use shards like that for spot data. And if you want to do a bit more, getting an excel sheet to populate with data from shards is on the more basic end of stuff to learn.

I can pull it in for nations one at a time, but if I just let the excel sheet run itself it would pull in way faster than the API limit. I don't know how to slow an excel sheet down, I've never needed to make Excel work less fast before.

[Imperium Anglorum]

I've never needed to make Excel work less fast before.

When has anyone ever need Excel to work less fast?

[Topid]

I've never needed to make Excel work less fast before.

When has anyone ever need Excel to work less fast?

When your formula pulls info from the API so you don't want it to make 300 calls all at once when you paste nation names into the input column.

Not the most well-versed on the API rules but I'm assuming 300 cells updating in Excel counts as 300 API calls and doesn't need to happen in less than a second.

Pulling info for 300 cells (or more) from the data dump doesn't affect the API though, that's why I thought I'd ask if it was doable.

[Eluvatar]

When has anyone ever need Excel to work less fast?

When your formula pulls info from the API so you don't want it to make 300 calls all at once when you paste nation names into the input column.

Not the most well-versed on the API rules but I'm assuming 300 cells updating in Excel counts as 300 API calls and doesn't need to happen in less than a second.

Pulling info for 300 cells (or more) from the data dump doesn't affect the API though, that's why I thought I'd ask if it was doable.

If you want simple, this might work best.

[Topid]

When your formula pulls info from the API so you don't want it to make 300 calls all at once when you paste nation names into the input column.

Not the most well-versed on the API rules but I'm assuming 300 cells updating in Excel counts as 300 API calls and doesn't need to happen in less than a second.

Pulling info for 300 cells (or more) from the data dump doesn't affect the API though, that's why I thought I'd ask if it was doable.

If you want simple, this might work best.

I'm actually backing into endorsements given to track and recognize good tarters. The sheet I built last night started with a list of WAs in #REGION which I pull from the data dump in a seperate tab. Then received the endorsement shard for each from the API, then filtered it so I ended up with a list for each I can work with in Excel. So far I've built it to count how many times each nation's name is in the entire list of endorsements which gives me how many nations they have endorsed (less any that weren't WAs or in #REGION at last update).

I've also already built it so that it checks if certain nations have endorsed each nation (this could be Delegate, Vice-Delegate, or any position that is supposed to tart) and outputs hyperlinks to the nations that the officer hasn't endorsed.

I'm looking into having it do more, but those are the core things I want it to do. As of now, it can all work just fine except that I am manually auto-filling the formula that pulls in the endorsement shard down one cell at a time so it doesn't do them all at once and block my access to the API. That's not really convient though, even if it is workable.

[Ever-Wandering Souls]

If you want simple, this might work best.

I'm actually backing into endorsements given to track and recognize good tarters. The sheet I built last night started with a list of WAs in #REGION which I pull from the data dump in a seperate tab. Then received the endorsement shard for each from the API, then filtered it so I ended up with a list for each I can work with in Excel. So far I've built it to count how many times each nation's name is in the entire list of endorsements which gives me how many nations they have endorsed (less any that weren't WAs or in #REGION at last update).

I've also already built it so that it checks if certain nations have endorsed each nation (this could be Delegate, Vice-Delegate, or any position that is supposed to tart) and outputs hyperlinks to the nations that the officer hasn't endorsed.

I'm looking into having it do more, but those are the core things I want it to do. As of now, it can all work just fine except that I am manually auto-filling the formula that pulls in the endorsement shard down one cell at a time so it doesn't do them all at once and block my access to the API. That's not really convient though, even if it is workable.

This may be silly, but - can you paste it into 50 cells, wait thirty seconds, and repeat? Rate limit is technically 50calls/30seconds, and if the formula is the same....

Still have to do it manually, but a easier than one at a time.

[Topid]

I'm actually backing into endorsements given to track and recognize good tarters. The sheet I built last night started with a list of WAs in #REGION which I pull from the data dump in a seperate tab. Then received the endorsement shard for each from the API, then filtered it so I ended up with a list for each I can work with in Excel. So far I've built it to count how many times each nation's name is in the entire list of endorsements which gives me how many nations they have endorsed (less any that weren't WAs or in #REGION at last update).

I've also already built it so that it checks if certain nations have endorsed each nation (this could be Delegate, Vice-Delegate, or any position that is supposed to tart) and outputs hyperlinks to the nations that the officer hasn't endorsed.

I'm looking into having it do more, but those are the core things I want it to do. As of now, it can all work just fine except that I am manually auto-filling the formula that pulls in the endorsement shard down one cell at a time so it doesn't do them all at once and block my access to the API. That's not really convient though, even if it is workable.

This may be silly, but - can you paste it into 50 cells, wait thirty seconds, and repeat? Rate limit is technically 50calls/30seconds, and if the formula is the same....

Still have to do it manually, but a easier than one at a time.

Oh really? Yeah that's doable. I've always heard them say 1 call per .6 seconds.

[Ever-Wandering Souls]

This may be silly, but - can you paste it into 50 cells, wait thirty seconds, and repeat? Rate limit is technically 50calls/30seconds, and if the formula is the same....

Still have to do it manually, but a easier than one at a time.

Oh really? Yeah that's doable. I've always heard them say 1 call per .6 seconds.

They're on one level all the same thing, but the API Doc puts it as "API Rate Limit: 50 requests per 30 seconds." (with a 15 min ban for exceeding it, so be careful) - if you're doing many at a continuous interval, then yes, 1 per .6 seconds is the minimum to stay within that limit. But if it's more beneficial to your purposes to work it batches, that's where 50/30 comes into play.

(Additionally, if you're trying to figure out how long something will take without a calculation, 100/min helps, but that's not a burst rate you can actually use in practice)

[Agadar]

I'm having trouble loading the data from the nations dump XML file into a hierarchy of maps and collections in Java. It seems to take forever. If anyone has any tips for me that they feel like sharing, then that would be swell.

Using JAXB, it still takes quite long to parse the nations dump file, but not nearly as long as when using my own code that doesn't rely on external libraries. However, JAXB demands I feed it custom classes to parse the dump file to, which I no longer want because that makes maintaining my Java API wrapper a nightmare as I have to update those custom classes every time the NS API is updated.

[The Irinixian Protectorate]

Can I make a request that the Happenings data have a <Nation> and/or <NationID> element added so that the data retrived from a happenings request can be easily sorted via xml tags. Also, this would remove the need for developers to parse the @@nation name@@ out of the text data of the item when it's received, since that would no longer be necessary.

Thanks,

Irinix, Supreme Queen of Irinixia

[Roavin]

I'm having trouble loading the data from the nations dump XML file into a hierarchy of maps and collections in Java. It seems to take forever. If anyone has any tips for me that they feel like sharing, then that would be swell.

Using JAXB, it still takes quite long to parse the nations dump file, but not nearly as long as when using my own code that doesn't rely on external libraries. However, JAXB demands I feed it custom classes to parse the dump file to, which I no longer want because that makes maintaining my Java API wrapper a nightmare as I have to update those custom classes every time the NS API is updated.

Not sure what options are available in Java, but you want to use a different kind of XML Parser. For example, in PHP you can use DOMDocument, which reads an entire XML file and builds a full representation in memory. For the giant nation dump files, that will obviously take a long time. Alternatively, you can use XMLReader, which does not build a full representation in memory but rather is just a tool to step through all the elements in an XML file. That will, obviously, be much faster.

There's surely such a XML parser available in Java too.

[Agadar]

I'm having trouble loading the data from the nations dump XML file into a hierarchy of maps and collections in Java. It seems to take forever. If anyone has any tips for me that they feel like sharing, then that would be swell.

Using JAXB, it still takes quite long to parse the nations dump file, but not nearly as long as when using my own code that doesn't rely on external libraries. However, JAXB demands I feed it custom classes to parse the dump file to, which I no longer want because that makes maintaining my Java API wrapper a nightmare as I have to update those custom classes every time the NS API is updated.

Not sure what options are available in Java, but you want to use a different kind of XML Parser. For example, in PHP you can use DOMDocument, which reads an entire XML file and builds a full representation in memory. For the giant nation dump files, that will obviously take a long time. Alternatively, you can use XMLReader, which does not build a full representation in memory but rather is just a tool to step through all the elements in an XML file. That will, obviously, be much faster.

There's surely such a XML parser available in Java too.

The bottleneck seems to be the creation of the hierarchy of maps. When I parse everything to a single map, the entire dump file is done within seconds. However, when I attempt to parse it to a neat hierarchy of maps, it doesn't even complete within a minute. I didn't even bother measuring exactly how long it takes beyond that minute.

So, I'm guessing I could just parse it to a single map, however that means that all map keys are along the lines of 'NATIONS.NATION.AGADAR.ANIMAL'. It's not hard for consumers of my API wrapper to use that, but it is certainly less user-friendly than a hierarchy of maps, no?

EDIT: FYI I'm using the javax XMLStreamReader, which from what I gathered is just about the quickest way one can go through an XML file in Java.

[Eluvatar]

Don't return a map, return an iterator.

[Eluvatar]

Can I make a request that the Happenings data have a <Nation> and/or <NationID> element added so that the data retrived from a happenings request can be easily sorted via xml tags. Also, this would remove the need for developers to parse the @@nation name@@ out of the text data of the item when it's received, since that would no longer be necessary.

Thanks,

Irinix, Supreme Queen of Irinixia

This would be ambiguous for happenings involving two nations, i.e. @@eluvatar@@ endorsed @@testlandia@@.

[Agadar]

Don't return a map, return an iterator.

I'm not sure what you mean. Could you elaborate?

[Eluvatar]

I was suggesting that parseNationsDump() should not return some object implementing Map<NationData> but some object implementing Iterator<NationData>.

Looking more closely at what you're doing, however, I'm surprised that your XMLStreamReader can construct a single map as quickly as you say. How long is it taking, exactly?

There's nothing inherently friendlier about dump["Agadar"]["ANIMAL"] than dump["NATIONS.NATION#Agadar.ANIMAL"]. Certainly it'd be easy enough to make a wrapper that converts the first into the second. What I'd ask is what the performance of doing the necessary fetches from your larger map is.

In my experience, any method of actually parsing the xml dumps is going to be painfully slow, but stream parsing is orders of magnitude faster than parsing into a full tree. (The fastest solution I've found is using grep | sed or rg --replace.)

[Agadar]

I was suggesting that parseNationsDump() should not return some object implementing Map<NationData> but some object implementing Iterator<NationData>.

Looking more closely at what you're doing, however, I'm surprised that your XMLStreamReader can construct a single map as quickly as you say. How long is it taking, exactly?

There's nothing inherently friendlier about dump["Agadar"]["ANIMAL"] than dump["NATIONS.NATION#Agadar.ANIMAL"]. Certainly it'd be easy enough to make a wrapper that converts the first into the second. What I'd ask is what the performance of doing the necessary fetches from your larger map is.

In my experience, any method of actually parsing the xml dumps is going to be painfully slow, but stream parsing is orders of magnitude faster than parsing into a full tree. (The fastest solution I've found is using grep | sed or rg --replace.)

I don't think you quite understand. The bottleneck with building the hierarchy of maps is the creation of the tens of thousands of submaps. The new-keyword is quite expensive in that regard, as is evidenced by the fact that when using only a single map, the Nations dump file is parsed within mere seconds.

As for the iterator suggestion, I need to have the keys as well, hence why I'm using a Map. Otherwise, how can consumers of my wrapper discover where their requested data is located when the data isn't mapped to keys?

[Eluvatar]

What NS++'s backend does is it builds a couple database tables from the dump XML files. (Other code then fetches the desired pieces of data from those tables). This avoids allocating hundreds of thousands of Map objects.

[Greater Tern]

Bug report: for the current SC resolution, the "REPEALS_COUNCILD" field isn't being returned. It should return "207" -- pointing to the SC resolution being repealed.

[Greater Tern]

Bug report: for the current SC resolution, the "REPEALS_COUNCILD" field isn't being returned. It should return "207" -- pointing to the SC resolution being repealed.

The same thing seems to be happening to the current GA resolution -- REPEALS_COUNCILD isn't included in the XML returned (REPEALS_COUNCILD should be 385).

[HMS Unicorn]

Bug report

The MEMBERLOG shard of the World Assembly API seems to have stopped updating.

https://www.nationstates.net/cgi-bin/api.cgi?wa=1&q=memberlog

The last reported event is from November 1st. As far as I can tell, the last reported WA admission was even earlier than that, on October 26th. From then until November 1st, it seems to have been reporting only WA departures, before it stopped completely.

The issue doesn't go away if one uses wa=2,

https://www.nationstates.net/cgi-bin/api.cgi?wa=2&q=memberlog

---

As a note for others running into issues because of the above bug:

It is possible to get the same information using the HAPPENINGS shard of the World API with filter option "member".

https://www.nationstates.net/cgi-bin/api.cgi?q=happenings;filter=member