NS Stats - Trends, Correlations, Anti-Correlations, Oh My!

[Valentine Z]

NS Stats - Trends, Correlations, Anti-Correlations, Oh My!

Being given a green light from a Mod, I was told that this will be the best place for this sort of thing. With that said, let's get down to business.

So, NS Stats! Ahh, some of us don't like it, some of us use it, some of us add modifiers to it to better reflect our nation. With that said, one thing is for sure, that the NS Stats are interesting breeds, and sometimes, you have trends that are obvious, or surprising, or sometimes... actually none at all and might not be related.

And I am going to find these trends!

Join me as we go through each of the "X against Y" trends, and we will find out what kind of secret relationships, if any, these stats have.

-----

Data collection / Disclaimer:

NationStates API is used and the API rate limit followed. The rate limit is 50 requests per 30 seconds (1.6667 requests per second). Using Python, I managed to procure up .csv data from the API, at a rate of around 0.3 - 0.6 requests per second using some delay timers. The data is then processed through an Excel file where I sort, create charts, and the like. I might put in more data as I go along, but this seems like a pretty promising sample space!

-----

I hope that this will spark some interest in the world of NS Stats, even if it is not too useful to check your outcome. It's a pretty fun exercise for me. ^^

I will give some time to discuss on the posted trend, and when the time is right, or when I'm free, or when the discussion slowed down. If any of those happens, I will move to a new trend. Though, if there's interest, I will re-visit the previous trends with more data. After all, this will be a dynamic project!

Moving on to a different trend doesn't mean that the discussion for that trend is over; you can still talk about it, but just that there will be more talk on the most recent trend posted.

Let's get started!

[Valentine Z]

List of trends using the 1st Batch of Data. Accurate as of 16th to 18th of November 2019.

1st Trawl is accurate as of 16-17th of Nov 2019. Regions trawled through: 10000 Islands, Capitalist Paradise, Europe, Forest, NationStates, Psychotic Dictatorships, The Free Nations Region, Two Letter, Europeia. The Pacifics and other large regions are ignored due to the sheer size of data collection.

2nd Trawl is accurate as of 18th of November 2019, at 19:18. The Rejected Realms' nations are added, making it a total of a whopping n = 10 568.

Number of Data Points: 10 568 nations, 908 848 points across 86 stats. There's a shortcoming that it doesn't really indicate the frequency of the points, so.... if there are a crapton of actual non-puppet nations on, say, the coordinates (70,70), we will not know from the graph alone unless I said so. I will annotate from the data, ignoring raiding/defending/N-Day/Z-Day puppets that doesn't have much in the way of answering issues.

X vs Y.
#1-1: Authoritarianism vs Ideological Radicality
#1-2: Religiousness vs Scientific Advancement
#1-3: Corruption vs Political Freedom

-----

List of trends using the 2nd Batch of Data. Accurate as of 24th and 25th of December 2019.

The 1st Trawl of the 2nd Batch has been done as of 24th and 25th of December 2019. There are a whopping 19 526 nations this time, 87 stats, 2 values for each stat (Score and World Rank). As a result, there are a crazy amount of data this time to work with, at a staggering 3 397 524 data points (1 698 762 data points with just nations and stats, without world rank).

Regions included in this trawl, along with my puppets: 10000 Islands, Capitalist Paradise, Europe, Forest, NationStates, Osiris, Texas, The Free Nations Region, The North Pacific, The Secret Order, Two Letters.

#2-1: Niceness (X) vs Compassion (Y) - Requested by Angshire
#2-2: Defense Forces (Y) vs Pacifism (X) - Requested by Angshire
#2-3: Economy (Y) vs Economic Freedom (X) - Requested by Angshire
#2-4: Environmental Beauty (Y) vs Economic Freedom (X) - Requested by Aclion
#2-5: Wealth Gap (Y) vs Economic Freedom (X) - Requested by Aclion
#2-6: Average Income (Y) vs Wealth Gaps (X) - Requested by Angshire
#2-7: Culture (Y) vs Weather (X) - Requested by Angshire
#2-8: Crime (Y) vs Civil Rights (X) - Requested by Angshire
#2-9: Public Education (Y) vs Public Healthcare (X) - Requested by Angshire and Bormiar.
#2-10: Public Education (Y) vs Public Transport (X) - Requested by Angshire
#2-11: Public Healthcare (Y) vs Public Transport (X) - Requested by Angshire
#2-12: Scientific Advancement (Y) vs Human Development Index (X) - Requested by Bormiar
#2-13: Health (Y) vs Public Healthcare (X) - Requested by Bormiar and Angshire
#2-14: Public Education (Y) vs Intelligence (X) - Requested by Bormiar and Sacara
#2-15: Compassion (X) vs Inclusiveness (Y) vs Safety (Z) - Requested by Bormiar
#2-16: Taxation (Y) vs HDI (X) - Requested by East Gondwana
#2-17: Law Enforcement (Y) vs Crime (X) - Requested by Angshire
#2-18: Law Enforcement (Y) vs Defense Forces (X) - Requested by Angshire
#2-19: Crime (Y) vs Safety (X) - Requested by Angshire
#2-20: Arms Manufacturing (Y) vs Defense Forces (X) - Requested by Pangurstan
#2-21: Civil Rights (Y) vs Economic Freedom (X) - Requested by Angshire
#2-22: Political Freedom (Y) vs Economic Freedom (X) - Requested by Angshire
#2-23: Civil Rights (X) vs Political Freedom (Y), Economic Freedom (Z) - Impromptu from Valentine Z
#2-24: Economic Freedom (X) vs Corruption (Y) - Requested by Angshire
#2-25: Arms Manufacturing (X) vs Corruption (Y) - by Valentine Z
#2-26: Public Healthcare (X) vs Public Education (Y) vs Public Transportation (Z) - by Valentine Z
#2-27: Authoritarianism (X) vs Intelligence (Y) - Requested by Khoronzon
#2-28: Economic Freedom (X) vs Obesity (Y) - Requested by Racoda
#2-29: Book Publishing (X) vs Intelligence (Y) - Requested by Khoronzon
#2-30: Compassion vs Arms Manufacturing - Requested by Khoronzon
#2-31: Rudeness (X) vs Tourism (Y) - Requested by Khoronzon
#2-32: Scientific Advancement (X) vs Primitiveness (Y) - Requested by Leutria
#2-33: Environmental Beauty (X) vs Primitiveness (Y) - Requested by Bears Armed
#2-34: Influence (Y) vs WA Endorsement (X) - Requested by Apabeossie
#2-35: Influence (Y) vs Residency (X) - Requested by Apabeossie
#2-36: Influence (Y) vs Population (X) - Requested by Apabeossie

In-queue:
#2-37: Health (Y) & Recreational Drug Use (X) - Requested by Betelgeuse Alliance
#2-38: Compassion (Y) & Economic Freedom (X) - Requested by Betelgeuse Alliance
#2-39: Welfare (Y) Economic Freedom (X) - Requested by Betelgeuse Alliance
#2-40: Income Equality (Y) vs Economy (X) - Requested by Giovanniland
#2-41: Black Market (Y) vs Population (X) - Inspired by Silver Commonwealth

-----

List of trends using the 3rd Batch of Data.

Data file: 2021-02-(24-27).csv. That is, from 24th to 27th of February. Consists of 238851 nations, along with many, many, MANY data for me to play and demonstrate with.

#3-1: Health (Y) & Recreational Drug Use (X) - Requested by Betelgeuse Alliance
#3-2: Compassion (Y) & Economic Freedom (X) - Requested by Betelgeuse Alliance
#3-3: Welfare (Y) vs Economic Freedom (X) - Requested by Betelgeuse Alliance
#3-4: Income Equality (Y) vs Economy (X) - Requested by Giovanniland
#3-5: Black Market (Y) vs Population (X) - Inspired by Silver Commonwealth
#3-6: Employment (Y) vs Economic Freedom (X) - Requested by Aurhany

-----

List of trends using the 4th Batch of Data.

Data file: All Nations - 2021-06-15 194113 (Processed).csv. That is, from 15th to 18th of June. Consists of 232271 nations, along with many, many, MANY data for me to play and demonstrate with.

#4-1: Health (Y) vs Food Quality (X) - by Valentine Z
#4-2: Taxation (Y) vs Economic Freedom (X) - Inspired by Merconitonitopia
#4-3: Employment (Y) vs Economic Freedom (X) - Requested by Barlyy

-----

List of trends using the 5th Batch of Data.

Unfortunately, due to the extremely long hiatus I have taken, as well as with a combination of me being unable to find my old files and now using a different method of interpreting data, I am using the new data from this point onwards. My apologies for those who have been waiting for so long for me to do an analysis with the data set at that time.

Data file: All Nations - 2021-10-18 115424.txt. Extracted around 15th to 18th Oct. Consists of ~245465 nations, now with Policies inside.

#5-1: Employment (Y) vs Economy (X) - Requested by Barlyy
#5-2: Employment (Y) vs Corruption (X) - Requested by Barlyy
#5-3: Food Quality (Y) vs Patriotism (X) - Requested by Bears Armed
#5-4: Obesity (Y) vs Food Quality (X) - Suggested by Trotterdam

-----

List of trends using the 6th Batch of Data.

There is a crash nearing the end that resulted in me getting 214956 nations out of 303585. Then, using another smaller extraction, I was able to get the remaining 89624 nations. The total nation will always change thanks to a slew of CTEs and foundings/refoundings, but nonetheless, this should cover at least 99.9% of the nations.

Data file: All Nations - 2024-02-28 111820.txt. Extracted from 28th Feb to 9th March. Consists of 309530 nations, now with Policies (properly done and taking out the nasty commas) inside.

#6-1: Culture (Y) vs Food Quality (X) - Requested by Land Without Shrimp
#6-2: Business Subsidization (Y) vs Economy (X) - Requested by Betelgeuse Alliance
#6-3: Compassion (X) vs Niceness (Y) vs Inclusiveness (Z) - Requested by Umbratellus
#6-4: Black Market (Y) vs Economic Freedom (X) - Requested by Wizlandia
#6-4-A1: Black Market Per Capita (Y) vs Economic Freedom (X)
#6-5: Black Market (Y) vs Taxation (X) - Requested by Wizlandia
#6-5-A1: Black Market Per Capita (Y) vs Taxation (X)
#6-6: Culture (Y) vs Civil Rights (X) - Requested by Wizlandia
#6-7: Culture (Y) vs Economic Freedom (X) - Requested by Wizlandia
#6-8: Culture (Y) vs Political Freedom (X) - Requested by Wizlandia
#6-9: Ideological Radicality (Y) vs Scientific Advancement (X) - Requested by Kors
#6-10: Ideological Radicality (Y) vs Defense Forces (X) - Requested by Kors
#6-11: Government Size (Y) vs Number of Issues Answered (X) - Requested by Thorn1000
#6-12: Death Rate (Y) vs Culture (X) - Requested by Bears Armed
#6-13: Number of Top 1% Gold Badges (Y) vs Number of Issues Answered (X) - Requested by Giovanniland
#6-14: Political Apathy (Y) vs Political Freedom (X) - Requested by Entropan
#6-15: Employment (Y) vs Recreational Drug Use (X) - Requested by Socialismia

-----

Completed Tasks

Extract the % Expenditure of the nations (military, education, health, etc, etc) - Requested by Candlewhisper Archive - Done and ready to use!
Extract the % Leading Cases of Death (suicide in police custody, nuclear spill, animal attack, etc etc) - Done and ready!
Extract the % Industry Sector (Private, Black Market, State, Government) - Done and ready!
Add Founded Time and DateTime of Extraction, both in Unix/Epoch format. Should be in GMT +8 - Done!
List of Policies - everything about them: Category, Internal Code, Name, Description (especially tricky, two of them have commas in them) - Done!

-----

Intermissions

Intermission - Population Gain / Day vs Population Size.
Intermission - Normalized Population Gain / Day vs Pop Size for 62-100 million pop nations.
Intermission - Combined Population and Nation count of NationStates - ~2018 Q4 to 2022 Feb 13.
Intermission - Founded Time (Epoch) vs Population.
Intermission - The Wonderful BUT WEIRD World of Sym Log.
ကြားပြတ်ချိန် - အယူဝါဒ အစွန်းရောက်မှု (Y) နှင့် မထူးဆန်းမှု (X)
Intermission - Ideological Radicality (Y) vs Averageness (X).

[Valentine Z]

Possible Expansion Space.

[Valentine Z]

Now this is a rather interesting one, as you can see. The Authoritarianism (Auth.) seems to have some sort of an exponential curve as Ideological Radicality (IR). The individual data points would then suggest otherwise that the high IR goes both ways - the high IR also exists for little to no Auth. nations; on the other hand, as you can tell, there's a general trend of the higher the Auth., the larger the IR (or vice versa). One thing is for sure, is that you cannot seem to have a largely dictatorial nation with a very low IR. It has to go up.

There are 385 nations with IR > 30 AND Auth > 3000; 497 nations with IR > 30 AND Auth <= 3000.
There are 414 nations with IR > 30 AND Auth > 2000; 468 nations with IR > 30 AND Auth <= 2000.
There are 466 nations with IR > 30 AND Auth > 1000; 416 nations with IR > 30 AND Auth <= 1000.

So, this all depends on whether you count something to be authoritarian. As the cutoff for Milli-Stalins changes, you will see that both sides of the Authoritative debate has the same IR if you are too extreme. After all, you have radical ideas on both ends.

A dictatorship can be extreme, and so is an anarchy.

Another thing I notice is that there seems to be a gradual decrease on the lower bound of Auth as IR increases. You can see that as IR increases, minimum Auth from these nations go from ~500 to 0.

-----

What do you think? Do you fit in the common trend, or do you have a dictatorship that's pretty average (low IR) ? Or an Anarchy with low IR ?

-----

Edit: More data is added, this time from The Rejected Realms. Effectively, I have twice the number of data to work with. For this trend, it seems to be more or less the same.

[Valentine Z]

Ahh, the rather controversial one that got a Beta recently (002b). As a result, the Scientific Advancement (SA), has been bumped up quite significantly while not affecting Religiousness (Religion) that much. Or at least, the religiousness nations now have a bigger scientific advancement.

The statement / trend of "Scientific Advancement being anti-correlative with Religion" is over... at least, so it seems. Okay, it has actually went down, but let's look at the graph.

X-axis is Scientific Advancement, runs from -2000 to +2000. Linear.
Y-axis is Religiousness, runs from 1 to 100 000. Logarithmic.

One thing I noticed is that Religiousness does not seem to be able to run below 1. According to the census at the time of this writing, the lowest ever achieved is 1.05 PPH (Prayers Per Hour). The highest goes to Deep South Borland, at 162 149.12 PPH, but they are not inside the dataset; highest goes to Askatopia, at 89 454.63 PPH.

Anyway, back to the trend... how does it look like? It kind of looks like y = ± x^0.5. Interesting thing is that for a decently high scientific advancement, you can enjoy the best of both badges. That is, you can a pretty high scientific advancement with high religiousness combined. However, the extremely religious nations seems to get the other end of the SA spectrum.

Some nations are exceptions to the rule, having -500 SA, while having maybe 1 < x < 10 in religiousness / PPH.

Let's look at the linear graph, then:

Ahh, now there's a certain trend to it. Sure, most of the nations (at least in my dataset) are not too high with religion, but you can see that there's a general decrease in SA as the religiousness goes way too high. On the other hand, high SA usually means moderately religious. The recent Beta didn't boost things too high, but a combination of high SA + high religion are still outliers.

Filtering the results, 550 nations are above 500 for Sci. Adv., that is, SA > 500, these are the results:
99 nations out of this 550, have > 100 PPH.
451 nations out of this 550, have < 100 PPH.
I cut it off at 12 PPH because between these nations, we went from 190.30 PPH to 12 PPH. I take > 100 PPH as being religiousness, but as with #1, different cutoffs give different results.

My own conclusion and theory? High Scientific Advancement and High Religiousness is possible, but it gets harder as your SA goes beyond 1400 - 1500. Extremely extremely highly religiousness nations are in the deep negative end of SA, and this will take some time through a balance of answering issues.

-----

What do you think?

Is your nation highly advanced in terms of science? Are you highly religiousness as well? Or is your nation very secular and atheist, and at the same time, very primitive?
(Also want to say that you guys are allowed to post here, if you thought otherwise. Maybe there was a lack of interest for the time being, but yeah, I have no thread ownership in GI, and you can talk here with me. ^^)

[The Discipled]

Delay timers (or rather the thread/sleep collections class) are relatively inefficient, i would have recommended implementing time complexity for nation polling gathering purposes. You can set the delay to that precise mathematical division of 1.6(>how long it is by internal decimal) by doing so, which will increase your total collated information.

[Valentine Z]

Delay timers (or rather the thread/sleep collections class) are relatively inefficient, i would have recommended implementing time complexity for nation polling gathering purposes. You can set the delay to that precise mathematical division of 1.6(>how long it is by internal decimal) by doing so, which will increase your total collated information.

That is true, since the gathering rate is a lot slower than the legal rate, so... I guess I can see to that later. ^^ Because the time taken also involves appending to the .csv file, so I am not sure how long it takes to trawl on the API itself, while not accounting for the data appends.

EDIT: I tested with 52 nations without the sleep timer, and it took 50.67 seconds (avg. 1.02 requests / sec), so... I would imagine that the append and reading the XML data themselves are longer. I suppose I could lower the sleep timer, just to make sure that I don't take chances in breaking the API limit.

[The Discipled]

Delay timers (or rather the thread/sleep collections class) are relatively inefficient, i would have recommended implementing time complexity for nation polling gathering purposes. You can set the delay to that precise mathematical division of 1.6(>how long it is by internal decimal) by doing so, which will increase your total collated information.

That is true, since the gathering rate is a lot slower than the legal rate, so... I guess I can see to that later. ^^ Because the time taken also involves appending to the .csv file, so I am not sure how long it takes to trawl on the API itself, while not accounting for the data appends.

EDIT: I tested with 52 nations without the sleep timer, and it took 50.67 seconds (avg. 1.02 requests / sec), so... I would imagine that the append and reading the XML data themselves are longer. I suppose I could lower the sleep timer, just to make sure that I don't take chances in breaking the API limit.

It would indeed, since the timer is not only waiting for the response but also the updating of the XML file. This can be solved by having the XML file update in buffers, and utilising a data structure (say an Arraylist) for storing recent statistical changes. This allows you to set the delay timer to begin to poll once there is a free allocation in memory instead of waiting for the XML file to update.

This will allow you to poll and update the XML file at the same time (asynchronous execution), as well as solve your problem of the timer kicking in only on completion of the entire task.

[Marxist Germany]

Is your nation highly advanced in terms of science? Are you highly religiousness as well? Or is your nation very secular and atheist, and at the same time, very primitive?
(Also want to say that you guys are allowed to post here, if you thought otherwise. Maybe there was a lack of interest for the time being, but yeah, I have no thread ownership in GI, and you can talk here with me. ^^)

I am in the top 5% for scientific advancement and top 2% for religiousness, I guess I can say that it is possible to maintain a balance. Also, great idea!

[Xupqeu]

Is your nation highly advanced in terms of science? Are you highly religiousness as well? Or is your nation very secular and atheist, and at the same time, very primitive?
(Also want to say that you guys are allowed to post here, if you thought otherwise. Maybe there was a lack of interest for the time being, but yeah, I have no thread ownership in GI, and you can talk here with me. ^^)

Yes, I'm currently at #1 position for scientific advancement in my region, but my country is really secular, although only because my personal bias in answering issues.

I've seen some country with high scientific advancement and religosity, so I think having high stat in both is possible. But, I've never seen any country with high secularism and primitiveness stat.

Now, if only I can reduce the massive taxation in my country without reducing the scientific advancement...

[TalAkMaChen]

...
I've seen some country with high scientific advancement and religosity, so I think having high stat in both is possible. But, I've never seen any country with high secularism and primitiveness stat.

I've seen similar trends, e.g. in this card collection Back to the Primitive, where often you have the top stats on the cards, including top religiousness with primitiveness (and ignorance as common third high ranking stat).

[Valentine Z]

UPDATE on my personal project: As I realised, Betas have the potential to change and affect the trends in some manner sooner or later. And with a few Betas coming in, as well as more data from new census coming in, this will mean that I will be trawling through the data the second time. Or third... or fourth... n-th time.

However, the good news is that I have no problem with this. I will keep the old data, and if the Beta comes out for a particular stat, I will do potential revisits and see what has changed, or otherwise.

This is certainly a long-term project, and I am excited and looking forward to all the trends that we will find! ^^

Currently, I have 10 568 data points, and there are 168 952 nations as of the time of this writing, which effectively means 6.25 % of the data is represented. I could use a bigger sample space and bump it up to 10% in the future, which can be accomplished by visiting the biggest Pacific there is - The North Pacific.

That is true, since the gathering rate is a lot slower than the legal rate, so... I guess I can see to that later. ^^ Because the time taken also involves appending to the .csv file, so I am not sure how long it takes to trawl on the API itself, while not accounting for the data appends.

EDIT: I tested with 52 nations without the sleep timer, and it took 50.67 seconds (avg. 1.02 requests / sec), so... I would imagine that the append and reading the XML data themselves are longer. I suppose I could lower the sleep timer, just to make sure that I don't take chances in breaking the API limit.

It would indeed, since the timer is not only waiting for the response but also the updating of the XML file. This can be solved by having the XML file update in buffers, and utilising a data structure (say an Arraylist) for storing recent statistical changes. This allows you to set the delay timer to begin to poll once there is a free allocation in memory instead of waiting for the XML file to update.

This will allow you to poll and update the XML file at the same time (asynchronous execution), as well as solve your problem of the timer kicking in only on completion of the entire task.

Unfortunately, I will have to humbly admit that I am not that advanced just yet. I appreciate your input a lot, I really do, but I'm still a full-time student that is not too well-versed in his programming skills, I might add. I'm actually surprised that I got this far. ^^; I will take your words into consideration perhaps in the future to get the data more efficiently, but for this task, I think it will suffice.

[Flanderlion]

How accurate do you want the data? As the daily dumps have the census of the day.

[Valentine Z]

How accurate do you want the data? As the daily dumps have the census of the day.

The way I'm collecting is for every single stats of the nations in my list, so... I'm aware of the daily dumps, but those are not the complete set that I kinda want.

I figure that my method, even if slow, could be worth it for that few moments because those are the snapshots of the stats for that particular day, not waiting for the census of the day and missing out several days of progress.

[Merconitonitopia]

I would like to see the relationship between corruption and political freedom.

...
I've seen some country with high scientific advancement and religosity, so I think having high stat in both is possible. But, I've never seen any country with high secularism and primitiveness stat.

I've seen similar trends, e.g. in this card collection Back to the Primitive, where often you have the top stats on the cards, including top religiousness with primitiveness (and ignorance as common third high ranking stat).

I can see my house from here!
There are a few secular primitives, but they are few and far between. Domestic Corporate America is top 1 percent for both primitiveness and secularism.
It appears that almost all primitive nations are highly religious.

I don't know if there are any clever primitives. It seems that ignorance and primitiveness always go hand in hand. This is true even of primitive countries who top the ranks in education, like Gynostan.

[Bears Armed]

I don't know if there are any clever primitives. It seems that ignorance and primitiveness always go hand in hand. This is true even of primitive countries who top the ranks in education, like Gynostan.

Bears Armed: Top 0.2% for Primitive, Bottom 01% for Ignorance...
https://www.nationstates.net/nation=bea ... etail=rank

But maybe it's only Bears, and not Humans as well, that can manage this?

[Valentine Z]

As requested, here is the trend for Corruption and Political Freedom. So... these two stats. You have classifiers and categories for Political Freedom where it could go from World Benchmark to Corrupted at times. Do you always wonder if 100% political freedom can mean that your country is corrupted? Or was this another one of those general trends again? Let's find out!

Political Freedom: x-axis, Linear.
Corruption: y-axis, Logarithmic.

Political Freedom: x-axis, Linear.
Corruption: y-axis, Linear.

Political Freedom: x-axis, Log. Due to the implications of -Infinity with log(0.00), 0.00 Political Freedom is defaulted to 0.01 for this very instance.
Corruption: y-axis, Log.

So, you can see the general trend once again! As your political freedom increases, your corruption will decrease exponentially. That is, from the first graph, as Political Freedom moves to the right, the corruption follows a "linear" trend on the log graph, so it is actually decreasing exponentially. So far, so good, of course! Then it sort of just shot up again, with 100.00 Political Freedom countries just not getting the "Low Corrupt, High Integrity" attribute.

Then you see that there are also points and exceptions to the rule, as always. Some nations still have a high amount of corruption as political freedom increases, though not as bad as the dictatorships. An explanation can be that if you are too politically free, there's nothing stopping you from collecting bribes for governmental and bureaucratic stuff.

More comment on the data: There is a pool of "horizontal" data of 1.00 Corruption for a range of PF, which, in hindsight, could be due to the puppets that doesn't really move much in terms of issue-answering (TODO: this will be considered in my next trawl, to remove too many puppets from affecting the data too much). It's relatively difficult for me to gauge on the active nations to pick, as opposed to the non-active ones. There are those that live by the NS Stats, while there are other nations that don't really play by the stats since Day One. That is why you are seeing a lot of nations at the 1.38 Corruption mark, because there are many nations that don't answer issues and Corruption and Integrity is something that you will not get a badge at the beginning; you will have to earn it.

Using 1.38 Kickbacks Per Hour (KPH) as a mark, I will look at 1578 nations that have >= 80.00 Political Freedom score. Out of these 1578 nations:

• 1042 nations have < 1.38 KPH.
• 153 nations have exactly 1.38 KPH. The Queen's States, most of them. The nations from Two Letter, the region.
• 383 nations have > 1.38 KPH, and out of these nations, 98 of them are > 10 KPH, and 22 of them are > 100 KPH.
• Highest Political Freedom with Corruption from the dataset goes to Akabusi of The Rejected Realms as of the time of data collection, at 419.07 KPH for 100.00 Political Freedom and 81.00 Civil Rights. They are an Anarchy with World Benchmark Civil Rights, and Excessive Political Freedom.
• Highest Political Freedom (100.00) with Lowest Corruption goes to Transylvania of The Rejected Realms, all of this excluding the Two Letter Regions. 100 for both Civil and Political Rights/Freedom, while maintaining a cool 1.51 KPH. This nation never answered a single issue.

Still, you can see the general decrease as you move along, so there's that. A country can be politically free, but whether or not it's corrupt... is perhaps up to the persons.

-----

What do you think?

Does your anarchic nation have a crazily high corruption, or is your anarchy state actually a pretty honest one given the nature of the people?

[Valentine Z]

A few Betas have been implemented (namely the Weaponization and Economy)... I guess it's time for a new trawl.

Implemented Betas: https://www.nationstates.net/page=beta?archive=1

[Free Santa Rosa]

This is amazing. Can I suggest adding a trendline? I will also manifest that I'm used to looking at graph titles as (x-axis) vs. (y-axis) so I was kind of confused for a quick bit.

For a long-term thing, I think it would also be relevant to see how trends go across regions/region tags. Maybe there's a way to collate data across those lines as well?

[Valentine Z]

This is amazing. Can I suggest adding a trendline? I will also manifest that I'm used to looking at graph titles as (x-axis) vs. (y-axis) so I was kind of confused for a quick bit.

For a long-term thing, I think it would also be relevant to see how trends go across regions/region tags. Maybe there's a way to collate data across those lines as well?

Ahh, sure. I will consider adding a trendline (the one that fits the best) for the future analysis. I forgot to mention that one, sorry about that. My graphs, unless said otherwise, will almost always be Y against X. Now I am thinking of having four graphs as with #3 for better viewing - Linear vs Linear, Log vs Linear, and Linear vs Log, and Log vs Log. I think it will help with those results that are condensed inside.

I could do the regional thing, though... I'm not very sure how to get around doing that just yet. What I can do, however, is to analyse trends from the past with the current one, detailing the Betas that are implemented along the way. Because of the nature of the data I am collecting (every single stat from every single nation, not the regional average and stuff), it takes a lot of time to request that data under the limit and at the rate I am going at, collecting for ALL the nations across would take me ~ 2.5 days.

Also of note is that even the regional data (i.e. the nations inside of it) needs to be updated with every trawl because of the nature of nations CTE-ing/moving/getting kicked out/etc... As with 10000 Islands as an example:
- As of 18-11-2019, there were 1173 nations.
- As of 22-11-2019, there were 1168 nations.

I need to find a way to throw an error and continue with the collection if the code cannot find that particular data/nation.

-----

Additional note is that I might be away from a few days sometimes, a week even, owing to the busy schedule that I have with my school. ^^;

[Trotterdam]

Ahh, sure. I will consider adding a trendline (the one that fits the best) for the future analysis. I forgot to mention that one, sorry about that. My graphs, unless said otherwise, will almost always be Y against X. Now I am thinking of having four graphs as with #3 for better viewing - Linear vs Linear, Log vs Linear, and Linear vs Log, and Log vs Log. I think it will help with those results that are condensed inside.

I would suggest using rankings rather than actual values. That would avoid needing to worry about scaling shenanigans, since I'm pretty sure the game uses rather arbitrary scaling for some scores (Civil Rights and Political Freedom are "smooshed" using a piecewise-linear function, for example).

I've thought about this some because I've been considering doing my own correlation analysis for a while. It would be pretty easy to code, if I use the data I already have (i.e., skewed towards actively-issue-answering nations that aren't in the WA) rather than collecting it separately, but I've never found the motivation to actually put in the work to implement it, plus making graphs for every possible census score combination would take up a fair bit of disk space.

[Valentine Z]

Ahh, sure. I will consider adding a trendline (the one that fits the best) for the future analysis. I forgot to mention that one, sorry about that. My graphs, unless said otherwise, will almost always be Y against X. Now I am thinking of having four graphs as with #3 for better viewing - Linear vs Linear, Log vs Linear, and Linear vs Log, and Log vs Log. I think it will help with those results that are condensed inside.

I would suggest using rankings rather than actual values. That would avoid needing to worry about scaling shenanigans, since I'm pretty sure the game uses rather arbitrary scaling for some scores (Civil Rights and Political Freedom are "smooshed" using a piecewise-linear function, for example).

I've thought about this some because I've been considering doing my own correlation analysis for a while. It would be pretty easy to code, if I use the data I already have (i.e., skewed towards actively-issue-answering nations that aren't in the WA) rather than collecting it separately, but I've never found the motivation to actually put in the work to implement it, plus making graphs for every possible census score combination would take up a fair bit of disk space.

That itself is not a bad idea and I actually gave it some thought. I will check out the API to see if there's a data on rankings instead of stats. Though, this itself has a slight problem, I feel. For instance, the "distances" between the stats are more or less lost (like sometimes, there's a big number gap between 1st and 2nd for Pacifism), and other times, 100.00 Civil Rights nations all needed to be smoshed into "1st".

I kind of wanted to analyse the stat numbers themselves, but sure, I will see to it eventually. Like I said, I am planning this as a long-term project, so... let's see if my coding prowess improves and I can do more things efficiently. ^^

[Nouveau Yathrib]

List of trends using the 1st Batch of Data:
#1-1 - Authoritarianism vs Ideological Radicality
#1-2 - Religiousness vs Scientific Advancement
#1-3 - Corruption vs Political Freedom

Excellent work Valentine Z! This confirms a lot of what I've seen with my puppets that exist purely for the sake of being in/reaching specific WA govt categories.

1) more radical = more authoritarian, unless you're trying to become more radical by becoming an Anarchy

2) the most religious nations are very backwards, and the least religious nations tend to be rather advanced. But there are far more religious + advanced nations (Jamilkhuze, Syfenq, and many of my puppets fall into this category) than the reverse

3) it's possible to have low political freedoms without having super high corruption, but to get ultra-low levels of corruption you really need to be in the 70-80 range

[Trotterdam]

That itself is not a bad idea and I actually gave it some thought. I will check out the API to see if there's a data on rankings instead of stats.

Definitely. You just need to specify &mode=rank.

Alternatively, you can ask for the raw scores and compute the rankings yourself, ranking among all nations in your sample rather than all nations in the world. This would likely give you more digits of accuracy than the rankings provided by the server itself.

and other times, 100.00 Civil Rights nations all needed to be smoshed into "1st".

Yeah, figuring how to deal with that situation (there are several ways) is the biggest problem with using rankings.

[Valentine Z]

I got the one with the &mode=rank and as expected, I got back the ranks instead of the scores. Though, turns out.... I actually was getting the Ranks all this time; just that I didn't use them.

As for the scores? At least from the documentation, the ones that I am getting seem to be the raw values themselves.

For instance,

Code: Select all

https://www.nationstates.net/cgi-bin/api.cgi?nation=valentine_z;q=census;scale=all&mode=score

score: Raw value, according to the API Documentation.

-----

I wonder if I can get more precise data, like HDI to > 2 decimal places.