NationStates

Is there an archive of daily dumps somewhere or are they just overwritten completely?

Holykol wrote:Is there an archive of daily dumps somewhere or are they just overwritten completely?

I keep an archive on my computer (as does Ballotonia, I believe). NationStates does not.

I just may take the time to run some checks on my archive you tell me what to look for and why

I'm interested in graphing the world population (in millions of citizens) over a longer period of time. Would you be willing up upload the files?

Holykol wrote:I'm interested in graphing the world population (in millions of citizens) over a longer period of time. Would you be willing up upload the files?

Would a tarball of text files with all the national populations work for you? I don't want to uncompress all of the archived nations.xml files at once <_<

Yes that would surely work. Although, i'm not sure why you would need to uncompress them. Could they not be uploaded as one file? Or is it just too big and time consuming that way?

Can someone please upload the daily dump for June 28, 2012? I missed it and i need the file. Thanks. (nations)

Holykol wrote:Can someone please upload the daily dump for June 28, 2012? I missed it and i need the file. Thanks. (nations)

Uploaded here: http://www.filedropper.com/2012-06-28-nationsxml
EDIT: NB: this is a 137.0 MB download!
EDIT: Filedropper apparently ungzips the file without removing the .gz, and then proceeds to offer the uncompressed file for download as if it were a zipped file.

EDIT: Now using different file hosting: http://www.sendspace.com/file/iaz71f
This is a 15.0 MB download

Ballotonia

Hmm, i can normally open the daily dumps fine, but i can't open this one with either winrar or 7-zip, the file is corrupt or damaged or something?

Holykol wrote:Hmm, i can normally open the daily dumps fine, but i can't open this one with either winrar or 7-zip, the file is corrupt or damaged or something?

Did you notice the different file size? Apparently File Dropper unzip the file, and feeds back an UNCOMPRESSED file. 138 MB, utterly idiotic and a waste of bandwidth.

So, just remove the ".gz" from the filename and you're good to go.

EDIT: Now using different file hosting: http://www.sendspace.com/file/iaz71f
This is a 15.0 MB download

Ballotonia

Can someone please upload July 1, 2012 nation daily dump file too? Thanks.

*split off datadump requests from API thread*

Holykol wrote:Can someone please upload July 1, 2012 nation daily dump file too? Thanks.

Here: http://www.sendspace.com/file/80u3ez
NB: 14.8 MB download

I'm not gonna make a habit out of this. However, I'd like to make my entire Datadump repository available to all. What I need is a (free) place to store and serve them. Suggestions are welcome. Note that sendspace won't work cause their free storage removes all files after 30 days. Others require an account for any download at all.

Ballotonia

Box.com offers 5GB free storage and folders can be shared via link. Dropbox.com allows for public folders as well, and I currently have 5.75 GB (not sure how much you start out with, but you get extra space for referrals, etc. up to 16GB). A thought.

5GB would hold about 340 Nation Data Dumps.

Free hosting + a quick and dirty script to provide links to files + FTP to add new files.

All easily done. All quickly done, unless you count the time to upload all the dumps.


EDIT:
and another script to automatically get a new daily dump from NS once a day. Makes the FTP obsolete once old ones are uploaded.

FINE >_>
I'll make it tomorrow if nobody else posts and tells me not to before I do it.

For anyone who has the old dumps, I can give you FTP access to upload if you desire, or you can upload elsewhere and make me manually fetch them all

Would the people behind NS dossier not already have something like this already mapped? It might be worth having a word with them, if only to discount their method if it doesn't meet your requirements

Scoochi2 wrote:FINE >_>
I'll make it tomorrow if nobody else posts and tells me not to before I do it.

For anyone who has the old dumps, I can give you FTP access to upload if you desire, or you can upload elsewhere and make me manually fetch them all

Yay, a volunteer to handle the free hosting!

I got 6.26 GB worth of gzipped nation/region data dumps to contribute. Perhaps others have contributions too!

Ballotonia

Might I ask, where (URL) are the data dumps created? I might be able to create a script to obtain them.

More generally, see: http://www.nationstates.net/pages/api.html

Ballotonia

A free dropbox account starts off at 2GB and you get 500MB each referral up to 18GB, so with a little work its possible to store the files for free (unless you've already figured out another way).

Will someone please upload July 21 nation daily dump? Thanks.

Holykol wrote:I'm interested in graphing the world population (in millions of citizens) over a longer period of time. Would you be willing up upload the files?

FYI:

NSHistory graphs the population of The World with new entries weekly. Nation data goes back to May 15, 2003 and region data goes back to May 28, 2003. You can even export the raw data to Excel. If you use it elsewhere, just mention where you got it.

Ballotonia wrote:*split off datadump requests from API thread*

Holykol wrote:Can someone please upload July 1, 2012 nation daily dump file too? Thanks.

Here: http://www.sendspace.com/file/80u3ez
NB: 14.8 MB download

I'm not gonna make a habit out of this. However, I'd like to make my entire Datadump repository available to all. What I need is a (free) place to store and serve them. Suggestions are welcome. Note that sendspace won't work cause their free storage removes all files after 30 days. Others require an account for any download at all.

Ballotonia

Install apache on your computer.

Will someone please see how many lines there are in 6/20/2012 nation daily dump, or even how large the filesize is?

Can anyone post python code to get and unzip data dumps