NationStates API (nationdata/regiondata)

[Comfed]

What fun? There is already a website with all the issues results, and if you don't want spoilers for your issues... don't look for them?

..a website that is provided thanks to the efforts of player(s) (not staff: Trotterdam and those who've provided effect lines for him to track), and isn't guaranteed to be accurate (the way a NS API call would be). Trotterdam's tool is much closer to the spoiler thread in design than an API endpoint would be.

There's a world of difference between players maintaining potentially-accurate spoiler resources and NS just giving us the information whenever we ask.

I mean, if you don't want the spoilers, you can just not ask...

[[violet]]

Can we please get the SPDR score of each nation added to the daily dumps? It would avoid having to make dozens or hundreds of calls in order to check influence for all nations in a region.

Do you want each nation's Influence score in its current region only or in all regions?

[United Calanworie]

Can we please get the SPDR score of each nation added to the daily dumps? It would avoid having to make dozens or hundreds of calls in order to check influence for all nations in a region.

Do you want each nation's Influence score in its current region only or in all regions?

If it's possible to do without too much trouble, current region only would be best. If it's impractical to do that, all regions is fine.

[Roavin]

Oh having it in from all regions would absolutely be preferable, even though that might inflate the dump size quite a bit for active R/D puppets.

[United Calanworie]

Something out of the normal for me posting here in that I'm not going to push about daily dumps containing influence but I'd like to ask if it would be possible to have a dispatch edited event pushed via the happenings API. I had envisioned potentially building a search tool for dispatches using ElasticSearch as a backend, but keeping content up-to-date to be searched would be ideal, and unfortunately there is no good way to ensure that content is up to date other than fetching every single dispatch at the fastest rate possible... which with some back of the napkin math means that it would take anywhere from a week to two weeks (depending on how many IDs have actual dispatches associated with them now and don't just link to "sorry, but this dispatch no longer exists.") and somewhere in the ballpark of 1,008,000 to 1,730,740 API calls (as of this writing, it could be a few hundred more at the time of reading) in order to update all the dispatches.

Which is not ideal for either my program or the site being subjected to full-throttle API calls >_>

So if we could have a happenings event in the API as of current and/or an event added to the potential SSE API that would make life a fair bit easier for this project.

Also I guess there should possibly be a question of "is building the index going to be too much of a heavy load on the site and would you prefer that the queries be spanned across multiple weeks instead of running at 50/30 and/or should I abandon this idea before I break the site with calls" which is... probably a good question to ask when considering sending somewhere in the neighborhood of 1.7m API calls.

[[violet]]

Those API calls are likely to be quite lightweight, as they're just a simple db lookup, but the API ratelimit is really meant as a burst number, not a constant level of traffic for days or weeks, so it does scare me a bit.

It may be quicker and easier for me to set up an ElastiSearch backend here (or Sphinx, which is what we use elsewhere) than to deal with a couple million API calls plus add a new API service (ideally on a brand new SSE event server). But then again I'm not really able to add extra projects to the todo list right now, either. So I guess my answer for now is that those API calls, if you make them, should ideally run quite a bit slower than the maximum rate.

[Trotterdam]

https://www.nationstates.net/cgi-bin/api.cgi?q=banner;banner=t11 returns malformed XML:

Code: Select all

<WORLD>
<BANNERS>
<BANNER id="t11">
<NAME>Oversight</NAME>
<VALIDITY>Fund Law & Order in a low-tech nation</VALIDITY>
</BANNER></BANNERS>
</WORLD>

This should be escaped as "&" instead of "&".

(I don't believe it's currently relevant, but "<" would also need to be escaped if it appears anywhere.)

[United Calanworie]

Those API calls are likely to be quite lightweight, as they're just a simple db lookup, but the API ratelimit is really meant as a burst number, not a constant level of traffic for days or weeks, so it does scare me a bit.

It may be quicker and easier for me to set up an ElastiSearch backend here (or Sphinx, which is what we use elsewhere) than to deal with a couple million API calls plus add a new API service (ideally on a brand new SSE event server). But then again I'm not really able to add extra projects to the todo list right now, either. So I guess my answer for now is that those API calls, if you make them, should ideally run quite a bit slower than the maximum rate.

That's about what I figured. Is there any easy way (for you) to produce a dump file of the dispatches? I know size would probably be a pretty big concern considering that each dispatch could potentially be around 97kb in size and there are a lot of them. If that isn't possible, I guess I will have to reconsider how I build this index because I would like to make most of the information at least relatively up to date and not have dispatches that have been edited for the last month not be re-updated.

Thanks for the prompt answer though.

[[violet]]

https://www.nationstates.net/cgi-bin/api.cgi?q=banner;banner=t11 returns malformed XML

Fixed, thanks.

[SherpDaWerp]

At an indeterminate point in the future™, I might compile some data on issues author region residency. For the CTE'd ones, I'd like to know their CTE date so I only have to find specific dates of dumps instead of however many thousand dumps exist in total. Problem with that is, the boneyard is only accessible from the HTML site.

Is there any chance of adding a boneyard search endpoint that returns (at least) population, founding date, and last activity, for a specific nation? I'm not opposed to adding more information if it's easy (i.e. the lookup returns number of dispatches and card link already), but those three fields are all I need.

For ancient CTE'd nations, an array of data would be fine (i.e. same as dumps or whatever else), and maybe even an extra parameter "newest" could let people ask for only the most-recent boneyard entry for a nation.

[SherpDaWerp]

At an indeterminate point in the future™, I might compile some data on issues author region residency. For the CTE'd ones, I'd like to know their CTE date so I only have to find specific dates of dumps instead of however many thousand dumps exist in total. Problem with that is, the boneyard is only accessible from the HTML site.

Is there any chance of adding a boneyard search endpoint that returns (at least) population, founding date, and last activity, for a specific nation? I'm not opposed to adding more information if it's easy (i.e. the lookup returns number of dispatches and card link already), but those three fields are all I need.

For ancient CTE'd nations, an array of data would be fine (i.e. same as dumps or whatever else), and maybe even an extra parameter "newest" could let people ask for only the most-recent boneyard entry for a nation.

The thought occurs to me that this endpoint would be easily "abused" to create a refound bot, which isn't my intention. Even without an explicit "name availability" timestamp in the data, you could extrapolate based on population or founding date.

I guess the only way to get around that would be to only list the last activity date, which I guess I'm still OK with - it's better than nothing, and it would still let me work out the last-known-location of issues authors without downloading every dump.

[United Calanworie]

Do you want each nation's Influence score in its current region only or in all regions?

If it's possible to do without too much trouble, current region only would be best. If it's impractical to do that, all regions is fine.

One-month *bump* on this query. Something that I thought of would be influence returning as a list, like this for example:
<influence>
<region id="a region">124.0</region>
<region id="testregionia">2.0</region>
</influence>

This is obviously a change from my prior response of "current region only," but after thinking about it a bit more while I wrote this post, it seems more logical to provide all the data instead of a small subset.

[Balasai]

does data change like population leader name takes change in the api instantly or it update at time of data dump only

[Roavin]

does data change like population leader name takes change in the api instantly or it update at time of data dump only

Those update instantly.

[Eluvatar]

If it's possible to do without too much trouble, current region only would be best. If it's impractical to do that, all regions is fine.

One-month *bump* on this query. Something that I thought of would be influence returning as a list, like this for example:
<influence>
<region id="a region">124.0</region>
<region id="testregionia">2.0</region>
</influence>

This is obviously a change from my prior response of "current region only," but after thinking about it a bit more while I wrote this post, it seems more logical to provide all the data instead of a small subset.

This would be data we've never previously made public. What possible downsides would there be to making it public?

[United Calanworie]

One-month *bump* on this query. Something that I thought of would be influence returning as a list, like this for example:
<influence>
<region id="a region">124.0</region>
<region id="testregionia">2.0</region>
</influence>

This is obviously a change from my prior response of "current region only," but after thinking about it a bit more while I wrote this post, it seems more logical to provide all the data instead of a small subset.

This would be data we've never previously made public. What possible downsides would there be to making it public?

I'm not sure. We could potentially see automated "what region were you in?" scanner tools, but those can be done manually already. It's just a question of how much work somebody is willing to put in. R/D would definitely get a lot more *precise* with influence calculations, too, when all the information would be available in a daily dump, but as a counter-argument, that information is already (lump-sum) provided by the API, this would just be removing the API calls from the equation.

[Eluvatar]

Right, I'm specifically asking what undesirable outcomes there could be from revealing the specific amounts associated with specific regions. Yes it's possible to reconstruct a lot of that information from past daily dumps, but that is a much greater investment of effort.

[United Calanworie]

We could potentially see sleeper operations being blown with a minimum of effort for the suspicious parties, even potentially automatically. I'm sure there might be other implications as well, but I'm not particularly well-versed in how influence is used beyond for R/D operations.

[Eluvatar]

Yes this would be valuable for identifying what regions a recently-revived nation had been part of in the past. I'll leave this open here for a few days*, see if anyone comes out of the woodwork who thinks that would be a serious problem for them.

*At least a few days.

[Trotterdam]

Yes this would be valuable for identifying what regions a recently-revived nation had been part of in the past.

Does influence still decay while nations lie in the boneyard?

[Eluvatar]

Yes this would be valuable for identifying what regions a recently-revived nation had been part of in the past.

Does influence still decay while nations lie in the boneyard?

You may find this announcement relevant.

[[violet]]

New Region Shard: 'bannerurl', which is like 'banner' but requires less figuring out how to construct an image URL.

[Panagouge]

New Region Shard: 'bannerurl', which is like 'banner' but requires less figuring out how to construct an image URL.

Where exactly is this feature?

[[violet]]

Right here. Also documented in the usual place.

For handy comparison, here are the three banner-related region shards.

[Ferranghia]

Hi,

I've had a script running for the past three years which makes monthly reports on economic sectors and posts it as a dispatch on my nation.

All's been good until some weeks ago when for some reason the report was not posted. Fast forward to today (because I had no time to check) and, looking at the exception and census data, it seems there's a value that's being printed in the census XML in scientific notation, which the XML parser doesn't recognise.

The incriminating portion of the census is this

Code: Select all

<SCALE id="76">
<SCORE>4.271943e+15</SCORE>
<RANK>3442</RANK>
<RRANK>46</RRANK>
</SCALE>

Is this going to be fixed server side eventually, or should I start parsing the score manually?

Many thanks