NationStates API (nationdata/regiondata)

[Doge Land]

Market value shard for trading cards? I'm trying to display the MV of a bunch of cards in one place, and right now there's no way to do that

Never mind. I'm dumb. I JUST realized that there was a market value tag in the API.

[Rocain Founder]

I've discovered that using the API to get a census world rank for a nation which is very close to the bottom of the ranking can produce paradoxical results. Specifically, I am occasionally getting ranks greater than the number of nations returned by the World API a few seconds earlier. I'm guessing that this is due to the details of how stats for new nations get merged into the rankings. I also think there is likely a way around it, as I've not seen this happen with the rank information you can access via the nation page (though I have seen it happen on the World page, where I have been given a rank number past the last one that the GUI will allow me to display).

Is this a bug, or just an unavoidable result of the way things are done? (I suspect the latter.) Is there a way using the API for me to get an accurate rank, counting from the bottom (which is why I care about this anomaly)? I'm guessing there must be a way to get self-consistent results, since the nation page's rank seems to always produce such results (though maybe I've just been lucky not to see any failures).

If there's nothing I can do with the current API to get reasonable results for an inverted rank (this is, starting from the bottom and counting up), I'd like to suggest an enhancement to the census shard to get an "inverse rank" or, alternately, an enhancement to the numnations World shard to indicate that I want a value useful for each of a list of census-IDs, which is to say, the rank of the bottom nation for each census-ID.

Thanks.

[Trotterdam]

Well, you could retrieve all nations' census score data and then sort it yourself, rather than relying on the server's ranking. At the API's ratelimit, though, it would take you 37 hours to retrieve data on every nation in the world, so it wouldn't all be in sync. I'm pretty sure census scores weren't in the daily data dumps.

For perfect results, you'd want each individual nation's ranking to come with an "out of how many?" shard telling the number of nations at the time the ranking was last updated. Not only does this guarantee you get the most relevant value even if there's a difference between when nations get updated, but it also allows you to retrieve both the ranking and nation count in the same query, preventing even one-second race conditions.

It's not just the API that has this (or a similar) problem, though. Even if you just go to the world nation listing and click on the last page, you'll notice a bunch of empty pages at the end rather than being taken to the page containing the lowest-ranking nation.

[Rocain Founder]

Let me give a single illustrative example. The bottom nation for the Human Development Index ranking is Pencil Sharpeners (not my nation). When I checked right before starting this post, its rank was 222,225. But numnations was 222,258, and getting the rank of Pencil Sharpeners direct from its rank page for that census ID tells me it is number 222,225 out of 222,258, and so it seems to be at the 34th position from the bottom rather than the first. Earlier, before writing my first post, I had the opposite situation where my nation Rocain Reaper, which I believe to be 40 positions behind Pencil Sharpeners, was instead at a position past the last nation shown by the World mapping for HDI. (I presume that, at that moment, Pencil Sharpeners was also off the map, but I didn't think to check at that point.)

What I'd like is a way to work out that Rocain Reaper is the 41st worst (which I believe is the reality), but it sure looks like there's just no way to write API code to reliably determine that. The numbers above are bound to have changed slightly by the time I hit Submit on this post, but until I answer Rocain Reaper's next issue, I expect its true position of 41 from the bottom to remain constant (though obviously other unlikely events could also cause its position to legitimately change).

[Twertis]

A project of mine is made infinitely more complex and inefficient by the happenings shard’s 28-second delay. As I’m told, that delay is to make triggering more difficult. That’s fair, but triggering can only be done using the admin, change, embassy and cte filter types in the happenings shard (as well as no filter, ofc). All other filters (i.e. member, law, eject, rmb, dispatch, move, founding, vote, resolution, and endo) are bound to user actions rather than update, and therefore can’t be used to trigger. Would it be possible to remove the 28-second happenings delay when the filter contains no update-bound happenings (maybe just a matter of adding a conditional)?

[Merni]

A project of mine is made infinitely more complex and inefficient by the happenings shard’s 28-second delay. As I’m told, that delay is to make triggering more difficult. That’s fair, but triggering can only be done using the admin, change, embassy and cte filter types in the happenings shard (as well as no filter, ofc). All other filters (i.e. member, law, eject, rmb, dispatch, move, founding, vote, resolution, and endo) are bound to user actions rather than update, and therefore can’t be used to trigger. Would it be possible to remove the 28-second happenings delay when the filter contains no update-bound happenings (maybe just a matter of adding a conditional)?

move, member and endo are the happenings most commonly used by defenders (although mostly through the actual page rather than the API), so those would also have to be delayed.

[Indian andhra]

Can I get a python code example to get data dumps

[Indian andhra]

I am hosting a discord bot and giving private rmb commands to people in discord the headers of people are different but the ip got ratelimited why

[The Northern Light]

Bug report

The API call for all WA nations has exactly 10 nations listed with the first letter capitalized, when all other nations are listed with all letters in lower-case: https://www.nationstates.net/cgi-bin/ap ... &q=members

The offending nations are:

Becrines
Bogroygua
Braedal
CTLandia
JeffroCountry
Koobarat
Poppyroni
Tuplesh
UnionReich
Unitaryland

Not a hugely critical bug, but my understanding is that lists of nations in the API are always in lower-case. This also messes up the alphabetical ordering, as the capitalized nations are placed ahead of all others starting with a letter.

[[violet]]

Heads up: I fixed a bug that may have resulted in census scales 53 and 79 reporting non-scaled numbers when queried via the API. So if those numbers seem 1,000 larger or 10 times smaller respectively than before, that's why. The new values should match what is displayed via the HTML site.

[The Northern Light]

Bug report

The API call for all WA nations includes braedal, who was ejected from the WA several days ago.

[August]

Bug report

The API call for all WA nations includes braedal, who was ejected from the WA several days ago.

It is more than just that nation.

[Fluffiness]

I am trying to work with data dumps in Python. However, it seems I've got an encoding issue, which may be related to viewtopic.php?f=15&t=45424&p=20639464&hilit=daily+dump+python#p20639464

Using various xml python libraries (xml.sax, xmltodict, xml.dom, all reliying on the xml expat library). I have a

Code: Select all

xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1214, column 11


It looks like it's caused by the apostrophe in Surprise's

Didn't see that coming

motto. Indeed, if I remove it, the parsing proceeds further, until getting stuck by another issue of this kind.
At the top of the xml dump, it's written that it is utf-8, and the libraries should be able to handle it. However, doing the following seems to indicate the encoding may be something else :

Code: Select all

>>> import os   
>>> from chardet import detect
>>> with open("cardlist_S1.xml", 'rb') as f:
...     rawdata = f.read()
...     print(detect(rawdata)['encoding'])
...

Windows-1254

Has anyone another way to process data dumps in Python, or met this issue already ? Thank you for any answer

[Trotterdam]

It's not a Python issue. I checked https://www.nationstates.net/pages/cardlist_S1.xml.gz in a hex viewer and confirmed that it is, indeed, encoded in Windows-1252. In fact, it exclusively uses Windows-1252. When Unicode characters that are not representable in Windows-1252 appear, it does not encode them in UTF-8, but rather as XML character entities ("&#number;").

[PotatoFarmers]

Is there a possibility for the RMB search function be implemented as an api shard? Like some sort of ability for us to pull latest RMB messages, as well as track the last RMB messages of a particular user, via API instead of having to use the manual website way?

[Trotterdam]

An RMB search sounds like something that would be server-load-intensive, so it's probably not something we want to encourage people to do too frequently. Remember that you can still use the HTML site at lower ratelimits, and that you can add "/template-overall=none" to HTML requests to make them smaller and easier to parse.

[United Calanworie]

Is there a possibility for the RMB search function be implemented as an api shard? Like some sort of ability for us to pull latest RMB messages, as well as track the last RMB messages of a particular user, via API instead of having to use the manual website way?

There's always the messages shard in the region API. Then just save post IDs and check against your saved IDs whenever you call the API again.

[Trotterdam]

There's always the messages shard in the region API. Then just save post IDs and check against your saved IDs whenever you call the API again.

True. If you're only looking for recent posts, then first retrieving recent posts and then manually filtering them for whatever you're interested in is likely to be more efficient than first searching for posts meeting some criteria and then picking out the most recent ones.

[Elest Adra]

It seems like the `flag` shard now returns a link to an SVG file. My app doesn't handle SVGs well natively. Is it safe to transform these URLs to `png`?

[Trotterdam]

It seems like the `flag` shard now returns a link to an SVG file. My app doesn't handle SVGs well natively. Is it safe to transform these URLs to `png`?

I would assume that it simply returns the flag in whatever format the player chose to upload it. The game doesn't currently convert between different image formats anywhere else, so I wouldn't expect it to in the API.

You can probably find SVG-to-PNG converters somewhere, if you need to.

[Elest Adra]

It seems like the `flag` shard now returns a link to an SVG file. My app doesn't handle SVGs well natively. Is it safe to transform these URLs to `png`?

I would assume that it simply returns the flag in whatever format the player chose to upload it.

Maybe. Personally I chose a default flag for Elest Adra - the flag of Djibouti - and it used to be linked to as a PNG, now an SVG, unfortunately.

[Trotterdam]

Yeah, default flags were changed from PNG to SVG recently. The old PNG files (and even older JPEG files) are still on the server, but are no longer officially linked anywhere, so you have to make an educated guess as to their names, and they presumably won't get updated going forward.

Contrast:
https://www.nationstates.net/images/flags/Default.svg
https://www.nationstates.net/images/flags/Default.png
https://www.nationstates.net/images/flags/default.jpg
Note that the PNG version has a more orange circle than either the JPEG or SVG versions, and that the JPEG version has a squarer aspect ratio than the PNG and SVG versions. They're not simply interchangeable, they're different interpretations of the flag. Also note the case-sensitive filenames.

[Imperium Anglorum]

Removed GENSEC and HELD from the API since they're not ready for public consumption yet.

Could these be added back to the API?

[[violet]]

Relevant post for people here: viewtopic.php?f=15&t=515460

[Valkyriah]

After I got the client key,what I should do with?