Datadump file requests

by **Holykol** » Tue Jun 19, 2012 5:27 am

Is there an archive of daily dumps somewhere or are they just overwritten completely?

by **Zemnaya Svoboda** » Tue Jun 19, 2012 5:59 am

Holykol wrote:Is there an archive of daily dumps somewhere or are they just overwritten completely?

I keep an archive on my computer (as does Ballotonia, I believe). NationStates does not.

I just may take the time to run some checks on my archive you tell me what to look for and why

by **Holykol** » Tue Jun 19, 2012 6:18 am

I'm interested in graphing the world population (in millions of citizens) over a longer period of time. Would you be willing up upload the files?

by **Zemnaya Svoboda** » Tue Jun 19, 2012 8:01 am

Holykol wrote:I'm interested in graphing the world population (in millions of citizens) over a longer period of time. Would you be willing up upload the files?

Would a tarball of text files with all the national populations work for you? I don't want to uncompress all of the archived nations.xml files at once <_<

by **Holykol** » Tue Jun 19, 2012 8:07 am

Yes that would surely work. Although, i'm not sure why you would need to uncompress them. Could they not be uploaded as one file? Or is it just too big and time consuming that way?

by **Holykol** » Fri Jun 29, 2012 1:53 pm

Can someone please upload the daily dump for June 28, 2012? I missed it and i need the file. Thanks. (nations)

by **Ballotonia** » Fri Jun 29, 2012 4:55 pm

Holykol wrote:Can someone please upload the daily dump for June 28, 2012? I missed it and i need the file. Thanks. (nations)

Uploaded here: http://www.filedropper.com/2012-06-28-nationsxml
EDIT: NB: this is a 137.0 MB download!
EDIT: Filedropper apparently ungzips the file without removing the .gz, and then proceeds to offer the uncompressed file for download as if it were a zipped file.

EDIT: Now using different file hosting: http://www.sendspace.com/file/iaz71f
This is a 15.0 MB download

Ballotonia

by **Holykol** » Sat Jun 30, 2012 9:21 pm

Hmm, i can normally open the daily dumps fine, but i can't open this one with either winrar or 7-zip, the file is corrupt or damaged or something?

by **Ballotonia** » Sun Jul 01, 2012 12:02 am

Holykol wrote:Hmm, i can normally open the daily dumps fine, but i can't open this one with either winrar or 7-zip, the file is corrupt or damaged or something?

Did you notice the different file size? Apparently File Dropper unzip the file, and feeds back an UNCOMPRESSED file. 138 MB, utterly idiotic and a waste of bandwidth.

So, just remove the ".gz" from the filename and you're good to go.

EDIT: Now using different file hosting: http://www.sendspace.com/file/iaz71f
This is a 15.0 MB download

Ballotonia

by **Holykol** » Tue Jul 03, 2012 4:47 pm

Can someone please upload July 1, 2012 nation daily dump file too? Thanks.

by **Ballotonia** » Tue Jul 03, 2012 10:00 pm

*split off datadump requests from API thread*

Holykol wrote:Can someone please upload July 1, 2012 nation daily dump file too? Thanks.

Here: http://www.sendspace.com/file/80u3ez
NB: 14.8 MB download

I'm not gonna make a habit out of this. However, I'd like to make my entire Datadump repository available to all. What I need is a (free) place to store and serve them. Suggestions are welcome. Note that sendspace won't work cause their free storage removes all files after 30 days. Others require an account for any download at all.

Ballotonia

by **Severisen** » Tue Jul 03, 2012 11:48 pm

Box.com offers 5GB free storage and folders can be shared via link. Dropbox.com allows for public folders as well, and I currently have 5.75 GB (not sure how much you start out with, but you get extra space for referrals, etc. up to 16GB). A thought.

5GB would hold about 340 Nation Data Dumps.

by **Scoochi2** » Wed Jul 04, 2012 6:20 am

Free hosting + a quick and dirty script to provide links to files + FTP to add new files.

All easily done. All quickly done, unless you count the time to upload all the dumps.

EDIT:
and another script to automatically get a new daily dump from NS once a day. Makes the FTP obsolete once old ones are uploaded.

by **Scoochi2** » Wed Jul 04, 2012 12:12 pm

FINE >_>
I'll make it tomorrow if nobody else posts and tells me not to before I do it.

For anyone who has the old dumps, I can give you FTP access to upload if you desire, or you can upload elsewhere and make me manually fetch them all

by **Letoilenoir** » Wed Jul 04, 2012 3:17 pm

Would the people behind NS dossier not already have something like this already mapped? It might be worth having a word with them, if only to discount their method if it doesn't meet your requirements

by **Ballotonia** » Wed Jul 04, 2012 11:09 pm

Scoochi2 wrote:FINE >_>
I'll make it tomorrow if nobody else posts and tells me not to before I do it.

For anyone who has the old dumps, I can give you FTP access to upload if you desire, or you can upload elsewhere and make me manually fetch them all

Yay, a volunteer to handle the free hosting!

I got 6.26 GB worth of gzipped nation/region data dumps to contribute. Perhaps others have contributions too!

Ballotonia

by **Mewsland** » Thu Jul 05, 2012 10:31 am

Might I ask, where (URL) are the data dumps created? I might be able to create a script to obtain them.

by **Holykol** » Thu Jul 05, 2012 11:48 am

http://www.nationstates.net/pages/nations.xml.gz

by **Ballotonia** » Thu Jul 05, 2012 12:34 pm

More generally, see: http://www.nationstates.net/pages/api.html

Ballotonia

by **Holykol** » Fri Jul 06, 2012 7:34 am

A free dropbox account starts off at 2GB and you get 500MB each referral up to 18GB, so with a little work its possible to store the files for free (unless you've already figured out another way).

by **Holykol** » Fri Aug 03, 2012 5:25 pm

Will someone please upload July 21 nation daily dump? Thanks.

by **NewTexas** » Fri Aug 03, 2012 9:26 pm

Holykol wrote:I'm interested in graphing the world population (in millions of citizens) over a longer period of time. Would you be willing up upload the files?

FYI:

NSHistory graphs the population of The World with new entries weekly. Nation data goes back to May 15, 2003 and region data goes back to May 28, 2003. You can even export the raw data to Excel. If you use it elsewhere, just mention where you got it.

:ugeek:

by **Linux and the X** » Sat Aug 04, 2012 5:11 am

Ballotonia wrote:*split off datadump requests from API thread*

Holykol wrote:Can someone please upload July 1, 2012 nation daily dump file too? Thanks.

Here: http://www.sendspace.com/file/80u3ez
NB: 14.8 MB download

I'm not gonna make a habit out of this. However, I'd like to make my entire Datadump repository available to all. What I need is a (free) place to store and serve them. Suggestions are welcome. Note that sendspace won't work cause their free storage removes all files after 30 days. Others require an account for any download at all.

Ballotonia

Install apache on your computer.

by **Holykol** » Fri Aug 10, 2012 6:48 am

Will someone please see how many lines there are in 6/20/2012 nation daily dump, or even how large the filesize is?

by **Indian andhra** » Sun Sep 19, 2021 4:14 am

Can anyone post python code to get and unzip data dumps

⚓ The North Pacific ⚓	Zemnaya Svoboda
☯ Taijitu ☯	Eluvatar
❂ Lemuria ❂	El Puppet

⚓ The North Pacific ⚓	Zemnaya Svoboda
☯ Taijitu ☯	Eluvatar
❂ Lemuria ❂	El Puppet