NATION

PASSWORD

NationStates API (nationdata/regiondata)

Bug reports, general help, ideas for improvements, and questions about how things are meant to work.

Advertisement

Remove ads

User avatar
Doge Land
Envoy
 
Posts: 279
Founded: Feb 15, 2019
Inoffensive Centrist Democracy

Postby Doge Land » Sun Aug 15, 2021 6:55 am

Market value shard for trading cards? I'm trying to display the MV of a bunch of cards in one place, and right now there's no way to do that

Never mind. I'm dumb. I JUST realized that there was a market value tag in the API.
Last edited by Doge Land on Sun Aug 15, 2021 11:04 am, edited 1 time in total.
its about drive its about power

delegate

User avatar
Rocain Founder
Spokesperson
 
Posts: 113
Founded: Aug 01, 2020
Father Knows Best State

Postby Rocain Founder » Sun Aug 22, 2021 11:18 am

I've discovered that using the API to get a census world rank for a nation which is very close to the bottom of the ranking can produce paradoxical results. Specifically, I am occasionally getting ranks greater than the number of nations returned by the World API a few seconds earlier. I'm guessing that this is due to the details of how stats for new nations get merged into the rankings. I also think there is likely a way around it, as I've not seen this happen with the rank information you can access via the nation page (though I have seen it happen on the World page, where I have been given a rank number past the last one that the GUI will allow me to display).

Is this a bug, or just an unavoidable result of the way things are done? (I suspect the latter.) Is there a way using the API for me to get an accurate rank, counting from the bottom (which is why I care about this anomaly)? I'm guessing there must be a way to get self-consistent results, since the nation page's rank seems to always produce such results (though maybe I've just been lucky not to see any failures).

If there's nothing I can do with the current API to get reasonable results for an inverted rank (this is, starting from the bottom and counting up), I'd like to suggest an enhancement to the census shard to get an "inverse rank" or, alternately, an enhancement to the numnations World shard to indicate that I want a value useful for each of a list of census-IDs, which is to say, the rank of the bottom nation for each census-ID.

Thanks.

User avatar
Trotterdam
Powerbroker
 
Posts: 9560
Founded: Jan 12, 2012
Left-Leaning College State

Postby Trotterdam » Sun Aug 22, 2021 1:16 pm

Well, you could retrieve all nations' census score data and then sort it yourself, rather than relying on the server's ranking. At the API's ratelimit, though, it would take you 37 hours to retrieve data on every nation in the world, so it wouldn't all be in sync. I'm pretty sure census scores weren't in the daily data dumps.

For perfect results, you'd want each individual nation's ranking to come with an "out of how many?" shard telling the number of nations at the time the ranking was last updated. Not only does this guarantee you get the most relevant value even if there's a difference between when nations get updated, but it also allows you to retrieve both the ranking and nation count in the same query, preventing even one-second race conditions.

It's not just the API that has this (or a similar) problem, though. Even if you just go to the world nation listing and click on the last page, you'll notice a bunch of empty pages at the end rather than being taken to the page containing the lowest-ranking nation.

User avatar
Rocain Founder
Spokesperson
 
Posts: 113
Founded: Aug 01, 2020
Father Knows Best State

Postby Rocain Founder » Sun Aug 22, 2021 3:52 pm

Let me give a single illustrative example. The bottom nation for the Human Development Index ranking is Pencil Sharpeners (not my nation). When I checked right before starting this post, its rank was 222,225. But numnations was 222,258, and getting the rank of Pencil Sharpeners direct from its rank page for that census ID tells me it is number 222,225 out of 222,258, and so it seems to be at the 34th position from the bottom rather than the first. Earlier, before writing my first post, I had the opposite situation where my nation Rocain Reaper, which I believe to be 40 positions behind Pencil Sharpeners, was instead at a position past the last nation shown by the World mapping for HDI. (I presume that, at that moment, Pencil Sharpeners was also off the map, but I didn't think to check at that point.)

What I'd like is a way to work out that Rocain Reaper is the 41st worst (which I believe is the reality), but it sure looks like there's just no way to write API code to reliably determine that. The numbers above are bound to have changed slightly by the time I hit Submit on this post, but until I answer Rocain Reaper's next issue, I expect its true position of 41 from the bottom to remain constant (though obviously other unlikely events could also cause its position to legitimately change).

User avatar
Twertis
Bureaucrat
 
Posts: 59
Founded: Apr 07, 2019
Ex-Nation

Postby Twertis » Sun Sep 05, 2021 11:43 am

A project of mine is made infinitely more complex and inefficient by the happenings shard’s 28-second delay. As I’m told, that delay is to make triggering more difficult. That’s fair, but triggering can only be done using the admin, change, embassy and cte filter types in the happenings shard (as well as no filter, ofc). All other filters (i.e. member, law, eject, rmb, dispatch, move, founding, vote, resolution, and endo) are bound to user actions rather than update, and therefore can’t be used to trigger. Would it be possible to remove the 28-second happenings delay when the filter contains no update-bound happenings (maybe just a matter of adding a conditional)?

User avatar
Merni
Ambassador
 
Posts: 1654
Founded: May 03, 2016
Democratic Socialists

Postby Merni » Mon Sep 06, 2021 12:13 am

Twertis wrote:A project of mine is made infinitely more complex and inefficient by the happenings shard’s 28-second delay. As I’m told, that delay is to make triggering more difficult. That’s fair, but triggering can only be done using the admin, change, embassy and cte filter types in the happenings shard (as well as no filter, ofc). All other filters (i.e. member, law, eject, rmb, dispatch, move, founding, vote, resolution, and endo) are bound to user actions rather than update, and therefore can’t be used to trigger. Would it be possible to remove the 28-second happenings delay when the filter contains no update-bound happenings (maybe just a matter of adding a conditional)?

move, member and endo are the happenings most commonly used by defenders (although mostly through the actual page rather than the API), so those would also have to be delayed.
Last edited by Merni on Mon Sep 06, 2021 12:13 am, edited 1 time in total.
Donate your free time
The Labyrinth |
RIP Residency 3.5.16-18.11.21, killed by simplistic calculation | TGW Warden | GA Committee List
OOC by default | Admins: Please allow blocking WA TGs!
E -8.0 S -3.18
Wrote 532 GA
meth
I've noticed that everybody that is for abortion has already been born. — Ronald Reagan (eww)
When the people are being beaten with a stick, they are not much happier if it is called 'the People’s Stick.' — Mikhail Bakunin (to Karl Marx)
You're supposed to be employing the arts of diplomacy, not the ruddy great thumping sledgehammers of diplomacy. — Ardchoille
The West won the world not by the superiority of its ideas or values or religion [...] but rather by its superiority in applying organised violence. — Samuel P. Huntington (even he said that!)

User avatar
Indian andhra
Attaché
 
Posts: 80
Founded: May 01, 2021
Democratic Socialists

Postby Indian andhra » Sun Sep 12, 2021 5:24 am

Can I get a python code example to get data dumps

User avatar
Indian andhra
Attaché
 
Posts: 80
Founded: May 01, 2021
Democratic Socialists

Postby Indian andhra » Sun Sep 12, 2021 5:35 am

I am hosting a discord bot and giving private rmb commands to people in discord the headers of people are different but the ip got ratelimited why

User avatar
The Northern Light
Spokesperson
 
Posts: 155
Founded: Oct 10, 2014
Iron Fist Consumerists

Postby The Northern Light » Sun Sep 19, 2021 4:28 pm

Bug report

The API call for all WA nations has exactly 10 nations listed with the first letter capitalized, when all other nations are listed with all letters in lower-case: https://www.nationstates.net/cgi-bin/ap ... &q=members

The offending nations are:
Becrines
Bogroygua
Braedal
CTLandia
JeffroCountry
Koobarat
Poppyroni
Tuplesh
UnionReich
Unitaryland

Not a hugely critical bug, but my understanding is that lists of nations in the API are always in lower-case. This also messes up the alphabetical ordering, as the capitalized nations are placed ahead of all others starting with a letter.
Last edited by The Northern Light on Sun Sep 19, 2021 4:34 pm, edited 1 time in total.
Home of the WADP, Planet Eras, and the Constibillocode!
Join The North Pacific!
Where the democracy is strong, the debate robust, and the rum plentiful!

Forum | Discord Chat | Citizenship | Executive Staff | North Pacific Army | World Assembly Ministry | Roleplay | Trading Cards | Handbook

User avatar
[violet]
Site Admin
 
Posts: 15365
Founded: Antiquity

Postby [violet] » Wed Oct 13, 2021 8:54 pm

Heads up: I fixed a bug that may have resulted in census scales 53 and 79 reporting non-scaled numbers when queried via the API. So if those numbers seem 1,000 larger or 10 times smaller respectively than before, that's why. The new values should match what is displayed via the HTML site.

User avatar
The Northern Light
Spokesperson
 
Posts: 155
Founded: Oct 10, 2014
Iron Fist Consumerists

Postby The Northern Light » Tue Nov 09, 2021 12:00 pm

Bug report

The API call for all WA nations includes braedal, who was ejected from the WA several days ago.
Home of the WADP, Planet Eras, and the Constibillocode!
Join The North Pacific!
Where the democracy is strong, the debate robust, and the rum plentiful!

Forum | Discord Chat | Citizenship | Executive Staff | North Pacific Army | World Assembly Ministry | Roleplay | Trading Cards | Handbook

User avatar
August
Spokesperson
 
Posts: 184
Founded: Oct 07, 2011
Anarchy

Postby August » Wed Nov 10, 2021 1:21 pm

The Northern Light wrote:Bug report

The API call for all WA nations includes braedal, who was ejected from the WA several days ago.
It is more than just that nation.
|| AA Founder - Retired.

My Projects: AugustinAndroid (Server) | Augustin Alliance (Server) | NS Leaders (Server) | Tech suggestions | About me
I heard it was you / talkin' 'bout a world where all is free / it just couldn't be / and only a fool would say that...

User avatar
Fluffiness
Civil Servant
 
Posts: 6
Founded: Nov 06, 2017
Left-wing Utopia

Postby Fluffiness » Mon Nov 22, 2021 8:27 am

I am trying to work with data dumps in Python. However, it seems I've got an encoding issue, which may be related to viewtopic.php?f=15&t=45424&p=20639464&hilit=daily+dump+python#p20639464

Using various xml python libraries (xml.sax, xmltodict, xml.dom, all reliying on the xml expat library). I have a

Code: Select all
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1214, column 11


It looks like it's caused by the apostrophe in Surprise's
Didn't see that coming
motto. Indeed, if I remove it, the parsing proceeds further, until getting stuck by another issue of this kind.
At the top of the xml dump, it's written that it is utf-8, and the libraries should be able to handle it. However, doing the following seems to indicate the encoding may be something else :

Code: Select all
>>> import os   
>>> from chardet import detect
>>> with open("cardlist_S1.xml", 'rb') as f:
...     rawdata = f.read()
...     print(detect(rawdata)['encoding'])
...

Windows-1254


Has anyone another way to process data dumps in Python, or met this issue already ? Thank you for any answer
Fully insurrectionary luxury queer space anarchism
Fluffiness overview here

She/They
Economic Left/Right: -8.25
Social Libertarian/Authoritarian: -8.51

User avatar
Trotterdam
Powerbroker
 
Posts: 9560
Founded: Jan 12, 2012
Left-Leaning College State

Postby Trotterdam » Mon Nov 22, 2021 8:53 am

It's not a Python issue. I checked https://www.nationstates.net/pages/cardlist_S1.xml.gz in a hex viewer and confirmed that it is, indeed, encoded in Windows-1252. In fact, it exclusively uses Windows-1252. When Unicode characters that are not representable in Windows-1252 appear, it does not encode them in UTF-8, but rather as XML character entities ("&#number;").

Previous

Advertisement

Remove ads

Return to Technical

Who is online

Users browsing this forum: Benevolent 1, Caphixia, Google Adsense [Bot], Maredita, The Assemblian, The Aveline Union, The Northwestern Bioregion, Unconquered Systems, Untecna

Advertisement

Remove ads