NATION

PASSWORD

Datadump file requests

Bug reports, general help, ideas for improvements, and questions about how things are meant to work.
User avatar
Holykol
Lobbyist
 
Posts: 19
Founded: Apr 14, 2012
Ex-Nation

NationStates API (nationdata/regiondata)

Postby Holykol » Tue Jun 19, 2012 5:27 am

Is there an archive of daily dumps somewhere or are they just overwritten completely?

User avatar
Zemnaya Svoboda
Diplomat
 
Posts: 828
Founded: Jan 06, 2004
Civil Rights Lovefest

Postby Zemnaya Svoboda » Tue Jun 19, 2012 5:59 am

Holykol wrote:Is there an archive of daily dumps somewhere or are they just overwritten completely?


I keep an archive on my computer (as does Ballotonia, I believe). NationStates does not.

I just may take the time to run some checks on my archive you tell me what to look for and why :)

User avatar
Holykol
Lobbyist
 
Posts: 19
Founded: Apr 14, 2012
Ex-Nation

Postby Holykol » Tue Jun 19, 2012 6:18 am

I'm interested in graphing the world population (in millions of citizens) over a longer period of time. Would you be willing up upload the files?

User avatar
Zemnaya Svoboda
Diplomat
 
Posts: 828
Founded: Jan 06, 2004
Civil Rights Lovefest

Postby Zemnaya Svoboda » Tue Jun 19, 2012 8:01 am

Holykol wrote:I'm interested in graphing the world population (in millions of citizens) over a longer period of time. Would you be willing up upload the files?


Would a tarball of text files with all the national populations work for you? I don't want to uncompress all of the archived nations.xml files at once <_<

260
1078
137
74
50
74
68
12
1394
2280
118
111
1412
293
12
5190
12
681
591
1404
575
1345
483
1437
1390
1424
2788
485
1410
1420
397
251
215
217
203
156
131
114
575
579
571
109
102
93
...

User avatar
Holykol
Lobbyist
 
Posts: 19
Founded: Apr 14, 2012
Ex-Nation

Datadump file requests

Postby Holykol » Tue Jun 19, 2012 8:07 am

Yes that would surely work. Although, i'm not sure why you would need to uncompress them. Could they not be uploaded as one file? Or is it just too big and time consuming that way?
Last edited by Holykol on Tue Jun 19, 2012 6:05 pm, edited 1 time in total.

User avatar
Holykol
Lobbyist
 
Posts: 19
Founded: Apr 14, 2012
Ex-Nation

Datadump file requests

Postby Holykol » Fri Jun 29, 2012 1:53 pm

Can someone please upload the daily dump for June 28, 2012? I missed it and i need the file. Thanks. (nations)
Last edited by Holykol on Fri Jun 29, 2012 1:53 pm, edited 2 times in total.

User avatar
Ballotonia
Site Admin
 
Posts: 5450
Founded: Antiquity
Liberal Democratic Socialists

Postby Ballotonia » Fri Jun 29, 2012 4:55 pm

Holykol wrote:Can someone please upload the daily dump for June 28, 2012? I missed it and i need the file. Thanks. (nations)


Uploaded here: http://www.filedropper.com/2012-06-28-nationsxml
EDIT: NB: this is a 137.0 MB download!
EDIT: Filedropper apparently ungzips the file without removing the .gz, and then proceeds to offer the uncompressed file for download as if it were a zipped file.

EDIT: Now using different file hosting: http://www.sendspace.com/file/iaz71f
This is a 15.0 MB download

Ballotonia
Last edited by Ballotonia on Sun Jul 01, 2012 12:14 am, edited 2 times in total.
"Een volk dat voor tirannen zwicht zal meer dan lijf en goed verliezen, dan dooft het licht…" -- H.M. van Randwijk

User avatar
Holykol
Lobbyist
 
Posts: 19
Founded: Apr 14, 2012
Ex-Nation

Postby Holykol » Sat Jun 30, 2012 9:21 pm

Hmm, i can normally open the daily dumps fine, but i can't open this one with either winrar or 7-zip, the file is corrupt or damaged or something?

User avatar
Ballotonia
Site Admin
 
Posts: 5450
Founded: Antiquity
Liberal Democratic Socialists

Postby Ballotonia » Sun Jul 01, 2012 12:02 am

Holykol wrote:Hmm, i can normally open the daily dumps fine, but i can't open this one with either winrar or 7-zip, the file is corrupt or damaged or something?


Did you notice the different file size? Apparently File Dropper unzip the file, and feeds back an UNCOMPRESSED file. 138 MB, utterly idiotic and a waste of bandwidth.

So, just remove the ".gz" from the filename and you're good to go.

EDIT: Now using different file hosting: http://www.sendspace.com/file/iaz71f
This is a 15.0 MB download

Ballotonia
Last edited by Ballotonia on Sun Jul 01, 2012 12:15 am, edited 1 time in total.
"Een volk dat voor tirannen zwicht zal meer dan lijf en goed verliezen, dan dooft het licht…" -- H.M. van Randwijk

User avatar
Holykol
Lobbyist
 
Posts: 19
Founded: Apr 14, 2012
Ex-Nation

Postby Holykol » Tue Jul 03, 2012 4:47 pm

Can someone please upload July 1, 2012 nation daily dump file too? Thanks.

User avatar
Ballotonia
Site Admin
 
Posts: 5450
Founded: Antiquity
Liberal Democratic Socialists

Postby Ballotonia » Tue Jul 03, 2012 10:00 pm

*split off datadump requests from API thread*

Holykol wrote:Can someone please upload July 1, 2012 nation daily dump file too? Thanks.


Here: http://www.sendspace.com/file/80u3ez
NB: 14.8 MB download

I'm not gonna make a habit out of this. However, I'd like to make my entire Datadump repository available to all. What I need is a (free) place to store and serve them. Suggestions are welcome. Note that sendspace won't work cause their free storage removes all files after 30 days. Others require an account for any download at all.

Ballotonia
"Een volk dat voor tirannen zwicht zal meer dan lijf en goed verliezen, dan dooft het licht…" -- H.M. van Randwijk

User avatar
Severisen
Chargé d'Affaires
 
Posts: 491
Founded: Mar 15, 2012
Father Knows Best State

Postby Severisen » Tue Jul 03, 2012 11:48 pm

Box.com offers 5GB free storage and folders can be shared via link. Dropbox.com allows for public folders as well, and I currently have 5.75 GB (not sure how much you start out with, but you get extra space for referrals, etc. up to 16GB). A thought.

5GB would hold about 340 Nation Data Dumps.
Last edited by Severisen on Tue Jul 03, 2012 11:50 pm, edited 4 times in total.
<mcmasterdonia> Ex-TBH dudes get all the NS girls.

Mallorea and Riva wrote:Yeah but no one here can read. Literacy is a tool used by fendas, like IRC or morals.

Sichuan Pepper wrote:Should I be worried that Mall said he wanted to invade my region?
Member of the Cult of the Overgoat.

Married to my best friends Xoriet and Astarial
Father to Ramaeus, Sylvia, Greyghost, Bachtendekuppen, Liliarchy, Jar, Cookie. (And Guy)
Former: Everything
Current: Nowhere man.
Past, Present, and Future: Nobody.


User avatar
Scoochi2
Envoy
 
Posts: 207
Founded: Antiquity
Ex-Nation

Postby Scoochi2 » Wed Jul 04, 2012 6:20 am

Free hosting + a quick and dirty script to provide links to files + FTP to add new files.

All easily done. All quickly done, unless you count the time to upload all the dumps.


EDIT:
and another script to automatically get a new daily dump from NS once a day. Makes the FTP obsolete once old ones are uploaded.
Last edited by Scoochi2 on Wed Jul 04, 2012 6:21 am, edited 1 time in total.
The problem with a UDP joke is that you have no idea if people got it.

User avatar
Scoochi2
Envoy
 
Posts: 207
Founded: Antiquity
Ex-Nation

Postby Scoochi2 » Wed Jul 04, 2012 12:12 pm

FINE >_>
I'll make it tomorrow if nobody else posts and tells me not to before I do it.

For anyone who has the old dumps, I can give you FTP access to upload if you desire, or you can upload elsewhere and make me manually fetch them all :p
The problem with a UDP joke is that you have no idea if people got it.

User avatar
Letoilenoir
Chargé d'Affaires
 
Posts: 424
Founded: Nov 26, 2010
Ex-Nation

Postby Letoilenoir » Wed Jul 04, 2012 3:17 pm

Would the people behind NS dossier not already have something like this already mapped? It might be worth having a word with them, if only to discount their method if it doesn't meet your requirements
KEEP THE BLOOD CAVE FREE

User avatar
Ballotonia
Site Admin
 
Posts: 5450
Founded: Antiquity
Liberal Democratic Socialists

Postby Ballotonia » Wed Jul 04, 2012 11:09 pm

Scoochi2 wrote:FINE >_>
I'll make it tomorrow if nobody else posts and tells me not to before I do it.

For anyone who has the old dumps, I can give you FTP access to upload if you desire, or you can upload elsewhere and make me manually fetch them all :p


Yay, a volunteer to handle the free hosting! :)

I got 6.26 GB worth of gzipped nation/region data dumps to contribute. Perhaps others have contributions too!

Ballotonia
"Een volk dat voor tirannen zwicht zal meer dan lijf en goed verliezen, dan dooft het licht…" -- H.M. van Randwijk

User avatar
Mewsland
Powerbroker
 
Posts: 8347
Founded: Nov 06, 2008
Inoffensive Centrist Democracy

Postby Mewsland » Thu Jul 05, 2012 10:31 am

Might I ask, where (URL) are the data dumps created? I might be able to create a script to obtain them.
The Ascendancy of Cártána
Any and all dates I use in MT and PMT RPs on the Diplomacy forums are on the Cártányne calendar. See this factbook for details.
I return at long last to P2TM: At the Gates of Elysium, the first part of a new planned series titled The Elysium Variations developed in collaboration with Zarkenis Ultima & Sovreignry, is open now!

User avatar
Holykol
Lobbyist
 
Posts: 19
Founded: Apr 14, 2012
Ex-Nation

Postby Holykol » Thu Jul 05, 2012 11:48 am


User avatar
Ballotonia
Site Admin
 
Posts: 5450
Founded: Antiquity
Liberal Democratic Socialists

Postby Ballotonia » Thu Jul 05, 2012 12:34 pm

More generally, see: http://www.nationstates.net/pages/api.html

Ballotonia
"Een volk dat voor tirannen zwicht zal meer dan lijf en goed verliezen, dan dooft het licht…" -- H.M. van Randwijk

User avatar
Holykol
Lobbyist
 
Posts: 19
Founded: Apr 14, 2012
Ex-Nation

Postby Holykol » Fri Jul 06, 2012 7:34 am

A free dropbox account starts off at 2GB and you get 500MB each referral up to 18GB, so with a little work its possible to store the files for free (unless you've already figured out another way).

User avatar
Holykol
Lobbyist
 
Posts: 19
Founded: Apr 14, 2012
Ex-Nation

Postby Holykol » Fri Aug 03, 2012 5:25 pm

Will someone please upload July 21 nation daily dump? Thanks.
Last edited by Holykol on Sat Aug 04, 2012 6:24 pm, edited 2 times in total.

User avatar
NewTexas
Spokesperson
 
Posts: 174
Founded: Antiquity
Civil Rights Lovefest

Postby NewTexas » Fri Aug 03, 2012 9:26 pm

Holykol wrote:I'm interested in graphing the world population (in millions of citizens) over a longer period of time. Would you be willing up upload the files?


FYI:

NSHistory graphs the population of The World with new entries weekly. Nation data goes back to May 15, 2003 and region data goes back to May 28, 2003. You can even export the raw data to Excel. If you use it elsewhere, just mention where you got it.

:ugeek:
Big Tex
Governor of Texas

Author: NSDossier

User avatar
Linux and the X
Negotiator
 
Posts: 5476
Founded: Apr 29, 2006
Ex-Nation

Postby Linux and the X » Sat Aug 04, 2012 5:11 am

Ballotonia wrote:*split off datadump requests from API thread*

Holykol wrote:Can someone please upload July 1, 2012 nation daily dump file too? Thanks.


Here: http://www.sendspace.com/file/80u3ez
NB: 14.8 MB download

I'm not gonna make a habit out of this. However, I'd like to make my entire Datadump repository available to all. What I need is a (free) place to store and serve them. Suggestions are welcome. Note that sendspace won't work cause their free storage removes all files after 30 days. Others require an account for any download at all.

Ballotonia

Install apache on your computer.
If you see I've made a mistake in my wording or a factual detail, telegram me and I'll fix it. I'll even give you credit for pointing it out, if you'd like.
BLUE LIVES MURDER

[violet]: Maybe we could power our new search engine from the sexual tension between you two.
Me, responding to a request to vote for a liberation: But... but that would blemish my near-perfect history of spitefully voting against anything the SC does!
Farnhamia: That is not to be taken as license to start calling people "buttmunch."

GPG key ID: A8960638 fingerprint: 2239 2687 0B50 2CEC 28F7 D950 CCD0 26FC A896 0638

they/them pronouns

User avatar
Holykol
Lobbyist
 
Posts: 19
Founded: Apr 14, 2012
Ex-Nation

Postby Holykol » Fri Aug 10, 2012 6:48 am

Will someone please see how many lines there are in 6/20/2012 nation daily dump, or even how large the filesize is?

User avatar
Indian andhra
Attaché
 
Posts: 76
Founded: May 01, 2021
Authoritarian Democracy

Postby Indian andhra » Sun Sep 19, 2021 4:14 am

Can anyone post python code to get and unzip data dumps

Next

Advertisement

Remove ads

Return to Technical

Who is online

Users browsing this forum: Pannamlam, Sulenia, United Republic Of The Grand Isle

Advertisement

Remove ads