NATION

PASSWORD

API Collaboration thread [SPLIT from sticky]

Bug reports, general help, ideas for improvements, and questions about how things are meant to work.
User avatar
Afforess
Ambassador
 
Posts: 1105
Founded: Jun 22, 2009
Ex-Nation

API Collaboration thread [SPLIT from sticky]

Postby Afforess » Tue Feb 05, 2013 2:23 pm

Laevendell wrote:And I'm working on a similar shard API for Java. Lots of choices! :)
It will be up on github or similar pretty soon, along with my Android app.


Want to throw me a link to your github project when it goes up? I'm a Java programmer by trade, would love to help out. I wrote a basic API in Java for telegram stuff a year ago, but haven't messed with it much since then. Curious to see how far you are. ;)

Laevendell wrote:Ah, that's nice to know. I'm guessing the time spent in a region isn't stored then though.

With a sufficient history of nation/region dumps, you could figure this out programmatically. I've had a script backing up region and nations xml dumps since June 24th last year, I'd be happy to share it.
Last edited by Afforess on Tue Feb 05, 2013 2:26 pm, edited 2 times in total.
Minister of the Interior, Capitalist Paradise

The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.

User avatar
Laevendell
Envoy
 
Posts: 220
Founded: Jan 29, 2011
Left-wing Utopia

Postby Laevendell » Tue Feb 05, 2013 2:36 pm

Afforess wrote:Want to throw me a link to your github project when it goes up? I'm a Java programmer by trade, would love to help out. I wrote a basic API in Java for telegram stuff a year ago, but haven't messed with it much since then. Curious to see how far you are. ;)

I think the link should be somewhere in this thread (probably not easy to find), but here it is: https://github.com/Limewood/NSDroid (previously at: https://code.google.com/p/ns-api/).
The Java API currently handles all regional shards and all national shards except the censuses. It hasn't got any WA or world shards yet.
I have a few additional methods in my Android project to handle posting on the RMB and answering issues that are not included in this API, since I decided to have only the actual NS API in that. There's an automatic check on the rate limit, so it won't be exceeded and a choice between URLConnection and HttpClient for connections. Feel free to use it or extend it if you like.

Laevendell wrote:Ah, that's nice to know. I'm guessing the time spent in a region isn't stored then though.

With a sufficient history of nation/region dumps, you could figure this out programmatically. I've had a script backing up region and nations xml dumps since June 24th last year, I'd be happy to share it.[/quote]
I'm sure Lordieth would be happy to hear that. I'm currently not planning to use time spent in a region for anything, but I think there can be very useful data in that dump history of yours for many ideas.
Last edited by Laevendell on Tue Jan 27, 2015 1:11 pm, edited 1 time in total.

User avatar
Afforess
Ambassador
 
Posts: 1105
Founded: Jun 22, 2009
Ex-Nation

Postby Afforess » Tue Feb 05, 2013 2:49 pm

Laevendell wrote:
Afforess wrote:Want to throw me a link to your github project when it goes up? I'm a Java programmer by trade, would love to help out. I wrote a basic API in Java for telegram stuff a year ago, but haven't messed with it much since then. Curious to see how far you are. ;)

I think the link should be somewhere in this thread (probably not easy to find), but here it is: https://code.google.com/p/ns-api/ (I went with Google Code and SVN instead, as I haven't quite figured out how to work around bugs in eGit).


I generally use TortoiseGit as it has a shallow learning curve coming from TortoiseSVN. Anyway, the choice is immaterial, I am familiar with and can use either SVN or Git. I like the MIT license... Is there perhaps a more appropriate forum to discuss this though?
Minister of the Interior, Capitalist Paradise

The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.

User avatar
Laevendell
Envoy
 
Posts: 220
Founded: Jan 29, 2011
Left-wing Utopia

Postby Laevendell » Tue Feb 05, 2013 3:28 pm

Afforess wrote:I generally use TortoiseGit as it has a shallow learning curve coming from TortoiseSVN. Anyway, the choice is immaterial, I am familiar with and can use either SVN or Git. I like the MIT license... Is there perhaps a more appropriate forum to discuss this though?

I don't know, but I guess a new thread could be started for the Java API (or a joint thread for all API's interfacing the NS API).

User avatar
Frisbeeteria
Game Moderator
 
Posts: 28021
Founded: Dec 16, 2003
Capitalizt

Postby Frisbeeteria » Tue Feb 05, 2013 3:40 pm

Laevendell wrote:I guess a new thread could be started for the Java API (or a joint thread for all API's interfacing the NS API).

Split from the sticky. Feel free to adjust the OP.

User avatar
Afforess
Ambassador
 
Posts: 1105
Founded: Jun 22, 2009
Ex-Nation

Postby Afforess » Tue Feb 05, 2013 4:11 pm

Laevendell wrote:
Afforess wrote:I generally use TortoiseGit as it has a shallow learning curve coming from TortoiseSVN. Anyway, the choice is immaterial, I am familiar with and can use either SVN or Git. I like the MIT license... Is there perhaps a more appropriate forum to discuss this though?

I don't know, but I guess a new thread could be started for the Java API (or a joint thread for all API's interfacing the NS API).


Okay, I am digging into the code. I have a ton of suggestions, please don't take them personal affront to your capabilities, they are just suggestions:


  • Ditch the singleton accessor. Provide a way to create a new instance of the main NSAPI, either by a factory, or a direct constructor. This will allow any application that uses the library to manage multiple instances of your api at a time. At the very least, I can't think of any times in my career when I've wished I used singletons more. Singletons are a language hack, IMHO.

  • Use Maven. It's not hard to learn. It makes dependencies easy. (PS. You are using outdated HTTPClient libs)

  • Either only use HTTPClient for connections or only use the Java network API, don't provide both options. From the looks, you seem to be trying to write a lightweight api, so mixing and matching heavy-weight libraries (HTTPClient) doesn't make much sense. I honestly don't have a preference to which you use, but consistency is important and will keep bugs down.

  • Don't extend Exception, extend RuntimeException. RuntimeExceptions are unchecked, so I don't need to catch them explicitly. All of the exceptions in your library aren't exactly cases where anything can be done anyway. Don't make the programmer write boilerplate if you can avoid it.

  • Don't throw checked exceptions [in public methods]. Throwing IOException, XMLParseExceptions, etc is the easy way to handle them, but it makes writing any code that actually does something with your library a huge PITA. Catch them, and wrap them in a runtime exception (see above point).
Last edited by Afforess on Tue Feb 05, 2013 4:17 pm, edited 3 times in total.
Minister of the Interior, Capitalist Paradise

The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.

User avatar
Laevendell
Envoy
 
Posts: 220
Founded: Jan 29, 2011
Left-wing Utopia

Postby Laevendell » Tue Feb 05, 2013 4:26 pm

Thanks for the split!

Afforess wrote:Ditch the singleton accessor. Provide a way to create a new instance of the main NSAPI, either by a factory, or a direct constructor. This will allow any application that uses the library to manage multiple instances of your api at a time. At the very least, I can't think of any times in my career when I've wished I used singletons more. Singletons are a language hack, IMHO.

The only reason I used a Singleton here is because of the NS API rate limit. If you construct more than one instance, you'll be able to go over the rate limit and then it would be useless.

Use Maven. It's not hard to learn. It makes dependencies easy. (PS. You are using outdated HTTPClient libs)

I have used maven in a few projects before, I just did this one rather quickly (as it's a free time project) and didn't find it necessary yet. In fact, it only had one dependency until recently. The outdated HTTPClient is due to Android compatibility.

Either only use HTTPClient for connections or only use the Java network API, don't provide both options. From the looks, you seem to be trying to write a lightweight api, so mixing and matching heavy-weight libraries (HTTPClient) doesn't make much sense. I honestly don't have a preference to which you use, but consistency is important and will keep bugs down.

I'm providing the HTTPClient option for Android developers, as URLConnection has bugs there (I don't actually include the libraries in the Android app, but use the standard ones in Android that correspond to the provided versions of the libraries). I kept the URLConnection option for people using plain Java as HTTPClient is apparently being phased out.

Don't extend Exception, extend RuntimeException. RuntimeExceptions are unchecked, so I don't need to catch them explicitly. All of the exceptions in your library aren't exactly cases where anything can be done anyway. Don't make the programmer write boilerplate if you can avoid it.

Don't throw checked exceptions [in public methods]. Throwing IOException, XMLParseExceptions, etc is the easy way to handle them, but it makes writing any code that actually does something with your library a huge PITA. Catch them, and wrap them in a runtime exception (see above point).

Oh, I'd actually forgotten about that, I can certainly fix those, thanks.

User avatar
Afforess
Ambassador
 
Posts: 1105
Founded: Jun 22, 2009
Ex-Nation

Postby Afforess » Tue Feb 05, 2013 4:33 pm

Laevendell wrote:Thanks for the split!

Afforess wrote:Ditch the singleton accessor. Provide a way to create a new instance of the main NSAPI, either by a factory, or a direct constructor. This will allow any application that uses the library to manage multiple instances of your api at a time. At the very least, I can't think of any times in my career when I've wished I used singletons more. Singletons are a language hack, IMHO.

The only reason I used a Singleton here is because of the NS API rate limit. If you construct more than one instance, you'll be able to go over the rate limit and then it would be useless.


Maybe. Maybe I'm running in a proxy (ok, bad example [I think it's against the ToS], but theoretically possible). I guess I don't see why it should be prohibited, especially since there is no code that actually requires the singleton. I also can see your point, but I err on the side of developer freedom.
Laevendell wrote:
Use Maven. It's not hard to learn. It makes dependencies easy. (PS. You are using outdated HTTPClient libs)

I have used maven in a few projects before, I just did this one rather quickly (as it's a free time project) and didn't find it necessary yet. In fact, it only had one dependency until recently. The outdated HTTPClient is due to Android compatibility.

Either only use HTTPClient for connections or only use the Java network API, don't provide both options. From the looks, you seem to be trying to write a lightweight api, so mixing and matching heavy-weight libraries (HTTPClient) doesn't make much sense. I honestly don't have a preference to which you use, but consistency is important and will keep bugs down.

I'm providing the HTTPClient option for Android developers, as URLConnection has bugs there (I don't actually include the libraries in the Android app, but use the standard ones in Android that correspond to the provided versions of the libraries). I kept the URLConnection option for people using plain Java as HTTPClient is apparently being phased out.


Ok, that's a good reason - but in that case, I would just support only HTTPClient. Then your library works correctly out-of-the-box on the desktop and Android, without confusing developers who wonder why that option is there. It also reduces your code-complexity and maintenance down the road. I'm not sure it matters that HTTPClient is no longer maintained, if it works for our use-cases, it doesn't matter whether it is supported or not. Worst-case, you can fork the project and manually provide any fixes you want.
Last edited by Afforess on Tue Feb 05, 2013 4:35 pm, edited 1 time in total.
Minister of the Interior, Capitalist Paradise

The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.

User avatar
Laevendell
Envoy
 
Posts: 220
Founded: Jan 29, 2011
Left-wing Utopia

Postby Laevendell » Wed Feb 06, 2013 4:28 am

I've implemented some of your proposed changes to the code and a new version (0.2.1) is now up on Google Code.
The exceptions are now RuntimeExceptions, the checked exceptions are wrapped in the same and the API only uses HTTPClient now, no choice of URLConnection. I also added some comments that I forgot last time around.

User avatar
Afforess
Ambassador
 
Posts: 1105
Founded: Jun 22, 2009
Ex-Nation

Postby Afforess » Wed Feb 06, 2013 6:11 am

Laevendell wrote:I've implemented some of your proposed changes to the code and a new version (0.2.1) is now up on Google Code.
The exceptions are now RuntimeExceptions, the checked exceptions are wrapped in the same and the API only uses HTTPClient now, no choice of URLConnection. I also added some comments that I forgot last time around.


Awesome. Now that the telegram API is partially documented, any thoughts on how to implement that?
Minister of the Interior, Capitalist Paradise

The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.

User avatar
Laevendell
Envoy
 
Posts: 220
Founded: Jan 29, 2011
Left-wing Utopia

Postby Laevendell » Wed Feb 06, 2013 7:03 am

Afforess wrote:Awesome. Now that the telegram API is partially documented, any thoughts on how to implement that?

Well, the mass telegram function isn't implemented yet, so that can't be tested at the moment.
I'm thinking a method sendTemplateTelegram(String nation, String telegramID, String secretKey), where nation is the recipient nation's id, telegramID is the id of the telegram template and secretKey is the secret key of said template.
I think it best to wait a little with implementing that though, since things could change and the rate limit for telegrams is not determined yet.

User avatar
Afforess
Ambassador
 
Posts: 1105
Founded: Jun 22, 2009
Ex-Nation

Postby Afforess » Wed Feb 06, 2013 1:20 pm

Laevendell wrote:
Afforess wrote:Awesome. Now that the telegram API is partially documented, any thoughts on how to implement that?

Well, the mass telegram function isn't implemented yet, so that can't be tested at the moment.
I'm thinking a method sendTemplateTelegram(String nation, String telegramID, String secretKey), where nation is the recipient nation's id, telegramID is the id of the telegram template and secretKey is the secret key of said template.
I think it best to wait a little with implementing that though, since things could change and the rate limit for telegrams is not determined yet.


Right, but we can still consider use cases. sendTelegram(nation, telegramId, secretKey) is logical, but I wonder what you think of other obvious cases, like sending telegrams to an entire region or all wa delegates. Is it best to make the application do that manually, since the NS shard API will need to be scraped to determine wa delegates and all region member nations? Also, will the telegram calls block after 20s (the wait limit currently is 3 per minute)? I don't know.

Also, I mavenized and forked a copy of your project to github, keeping all licenses and attributions to you. Just an FYI.
https://github.com/Afforess/ns-api
Minister of the Interior, Capitalist Paradise

The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.

User avatar
Lordieth
Post Czar
 
Posts: 31603
Founded: Jun 18, 2010
New York Times Democracy

Postby Lordieth » Wed Feb 06, 2013 2:51 pm

Is this for all API Collaborations? I don't know whether it's beneficial to discuss this here, or to start my own thread once I release the first version of SimpleNationAPI

I'm working on my own Nation API Script in PHP that I will also be releasing, and I welcome any input on its development and implementation.

At the moment it can create Nation objects, and access either the full API for the Nation, or any shards you specify, and store the returned API data in an array.

Next up I'll be working on methods to allow you to set rate limits for API access, as well as methods to download/unzip the daily data dump, and methods to parse it.

I'm also considering taking it one step further and including a set of function wrappers which essentially perform common tasks with the API, allowing those with less extensive experience with scripting to be able to use some pre-defined functions that will create the script for them with a minimum of changes required. Of course they won't be as flexible as the API itself, but they'll be purely optional for those who want to use them.
There was a signature here. It's gone now.

User avatar
Afforess
Ambassador
 
Posts: 1105
Founded: Jun 22, 2009
Ex-Nation

Postby Afforess » Wed Feb 06, 2013 4:26 pm

Lordieth wrote:Is this for all API Collaborations? I don't know whether it's beneficial to discuss this here, or to start my own thread once I release the first version of SimpleNationAPI


For all - or at least it can be.

Lordieth wrote:I'm working on my own Nation API Script in PHP that I will also be releasing, and I welcome any input on its development and implementation.


I'm not personally a fan of PHP, but I suppose an actual OO PHP API is better than mucking around in simple_xml parser and http queries.
Minister of the Interior, Capitalist Paradise

The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.

User avatar
Laevendell
Envoy
 
Posts: 220
Founded: Jan 29, 2011
Left-wing Utopia

Postby Laevendell » Wed Feb 06, 2013 4:32 pm

Afforess wrote:Right, but we can still consider use cases. sendTelegram(nation, telegramId, secretKey) is logical, but I wonder what you think of other obvious cases, like sending telegrams to an entire region or all wa delegates. Is it best to make the application do that manually, since the NS shard API will need to be scraped to determine wa delegates and all region member nations? Also, will the telegram calls block after 20s (the wait limit currently is 3 per minute)? I don't know.

Oh, very good ideas! I haven't had time to look at the new telegram system much due to (other) work, but I think it would be great to make some convenience methods for sending telegrams to the following groups:
* WA delegates (using the 'delegates' WA shard)
* WA members (using the 'members' WA shard)
* Nations in a region (using the 'nations' region shard)
* New nations (using the 'newnations' world shard)
Those lists of nations can each be fetched with one call to the API, so it's not very costly.
For more complicated actions, I think it would be better to let the developer manage it, since it's not obvious what they would find useful.

As for the telegram rate limit, I think it's going to be lower than 3 per minute at the start (if I understand the API comment correctly), but I believe [violet] stated before in the API thread that the NS server measures the number of calls over a 30 second time period for normal calls, so I guess it's probably the same here. I'm thinking using the same system as for the normal calls would be logical, but with a lower limit of course. I do believe the API call to send a template telegram would still count for the normal rate limit to, so that would have to be taken into account.

Also, I mavenized and forked a copy of your project to github, keeping all licenses and attributions to you. Just an FYI.
https://github.com/Afforess/ns-api

Very nice, thanks for the FYI. I have been considering removing the Singleton in favour of a public constructor, but with the ability to change the rate limit (so you could run several instances, each with a lower rate limit) or turn the check off (if you wanted to keep track of the limit yourself in a larger application). I'll probably copy over some of your changes to the SVN repo as well.

Lordieth wrote:Is this for all API Collaborations?

I think this thread is for all API collaboration projects, so feel free to post and share ideas and input. Perhaps there are more developers interested in contributing to the PHP API as well, or maybe to an API in another language completely (Prolog, anyone)?

User avatar
Afforess
Ambassador
 
Posts: 1105
Founded: Jun 22, 2009
Ex-Nation

Postby Afforess » Wed Feb 06, 2013 5:32 pm

Laevendell wrote:
Afforess wrote:R
For more complicated actions, I think it would be better to let the developer manage it, since it's not obvious what they would find useful.


Right. I am actually thinking of writing an application that looks at new nations, then examines their ideologies to see if they are say, Capitalist-leaning, then send them invites over to the Capitalist Paradise. Silly to waste my API limits on communist-kingdoms.

I've been messing around with the API, using the Play! framework for java/scala-based websites, seems to be really solid so far.
Minister of the Interior, Capitalist Paradise

The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.

User avatar
Afforess
Ambassador
 
Posts: 1105
Founded: Jun 22, 2009
Ex-Nation

Postby Afforess » Wed Feb 06, 2013 6:49 pm

I've added the world shards and world data to the ns-api, feel free to merge my changes: https://github.com/Afforess/ns-api/comm ... 52e2e028fc

I also added a verbose flag so I can see what is going on more easily. Helps to debug broken queries.
Last edited by Afforess on Wed Feb 06, 2013 6:49 pm, edited 1 time in total.
Minister of the Interior, Capitalist Paradise

The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.

User avatar
Laevendell
Envoy
 
Posts: 220
Founded: Jan 29, 2011
Left-wing Utopia

Postby Laevendell » Thu Feb 07, 2013 3:29 am

Afforess wrote:Right. I am actually thinking of writing an application that looks at new nations, then examines their ideologies to see if they are say, Capitalist-leaning, then send them invites over to the Capitalist Paradise. Silly to waste my API limits on communist-kingdoms.

Actually, that's very similar to what the recruitment application I wrote some time ago for my region (Democratic Socialist Assembly) does, except for democratic socialist nations (the socialist part is checked manually though, as it's hard to determine from stats). That application was written in PHP though, as Java servers are expensive and PHP servers. It keeps track of the API limit and also which nations were previously sent a recruitment telegram. It's not a pretty application, but it works. :)

Afforess wrote:I've been messing around with the API, using the Play! framework for java/scala-based websites, seems to be really solid so far.

That looks like a very interesting web framework! Some things look fairly similar to normal JavaEE development (and it seems you can use JPA and such), but the setup looks simpler, at least for websites and it seems to have many convenience functionalities.

Afforess wrote:I've added the world shards and world data to the ns-api, feel free to merge my changes: https://github.com/Afforess/ns-api/comm ... 52e2e028fc

I also added a verbose flag so I can see what is going on more easily. Helps to debug broken queries.

Very nice! I'll definitely merge that into the Google Play repo. You didn't add any licence info at the top of the new WorldData file though, who should I attribute the copyright to?

EDIT: I've added your code to the Google Code repo (https://code.google.com/p/ns-api/ for those reading this thread and haven't come from the sticky thread). I've also added some of the WA shards (numnations, numdelegates, delegates, members, happenings, memberlog and lastresolution), removed the Singleton, enabled changing the rate limit (from the default 49 per 30 secs) as well as disabling it (with a warning to manage it elsewhere).
Last edited by Laevendell on Thu Feb 07, 2013 6:08 am, edited 1 time in total.

User avatar
Afforess
Ambassador
 
Posts: 1105
Founded: Jun 22, 2009
Ex-Nation

Postby Afforess » Thu Feb 07, 2013 7:03 am

Laevendell wrote:
Afforess wrote:Right. I am actually thinking of writing an application that looks at new nations, then examines their ideologies to see if they are say, Capitalist-leaning, then send them invites over to the Capitalist Paradise. Silly to waste my API limits on communist-kingdoms.

Actually, that's very similar to what the recruitment application I wrote some time ago for my region (Democratic Socialist Assembly) does, except for democratic socialist nations (the socialist part is checked manually though, as it's hard to determine from stats). That application was written in PHP though, as Java servers are expensive and PHP servers. It keeps track of the API limit and also which nations were previously sent a recruitment telegram. It's not a pretty application, but it works. :)


Nice. I've only done some hacked up PHP that telegrams new nations that arrive to the Capitalist Paradise, not much else.

Laevendell wrote:
Afforess wrote:I've added the world shards and world data to the ns-api, feel free to merge my changes: https://github.com/Afforess/ns-api/comm ... 52e2e028fc

I also added a verbose flag so I can see what is going on more easily. Helps to debug broken queries.

Very nice! I'll definitely merge that into the Google Play repo. You didn't add any licence info at the top of the new WorldData file though, who should I attribute the copyright to?

EDIT: I've added your code to the Google Code repo (https://code.google.com/p/ns-api/ for those reading this thread and haven't come from the sticky thread). I've also added some of the WA shards (numnations, numdelegates, delegates, members, happenings, memberlog and lastresolution), removed the Singleton, enabled changing the rate limit (from the default 49 per 30 secs) as well as disabling it (with a warning to manage it elsewhere).


Ah sorry about that, I forgot to set up Eclipse to use your license copyright template. I fixed WorldData.java. It now has the same license header as the other files.

One of the other things I'm thinking about doing today is writing an OO representation of the regions and nations xml dumps, as they contain valuable information not in the shards. My only concern is that the sheer size of the xml data involved will make any sort of light-weight representation impossible - parsing the entire xml file once through, and then mapping the file to Java-based representations of the data is likely to produce thousands of objects. Similarly, keeping the xml in memory is also not ideal, with it being 5-30mb in size. Any other suggestions on a better way forward?
Last edited by Afforess on Thu Feb 07, 2013 7:05 am, edited 1 time in total.
Minister of the Interior, Capitalist Paradise

The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.

User avatar
Laevendell
Envoy
 
Posts: 220
Founded: Jan 29, 2011
Left-wing Utopia

Postby Laevendell » Thu Feb 07, 2013 11:26 am

Afforess wrote:Nice. I've only done some hacked up PHP that telegrams new nations that arrive to the Capitalist Paradise, not much else.

I've been considering doing something like that. Did you use a cron script to run it?

Afforess wrote:One of the other things I'm thinking about doing today is writing an OO representation of the regions and nations xml dumps, as they contain valuable information not in the shards. My only concern is that the sheer size of the xml data involved will make any sort of light-weight representation impossible - parsing the entire xml file once through, and then mapping the file to Java-based representations of the data is likely to produce thousands of objects. Similarly, keeping the xml in memory is also not ideal, with it being 5-30mb in size. Any other suggestions on a better way forward?

Well, to be easily accessible, but not take up memory, I'd probably have gone for a database, but I think we don't want to involve that in a light-weight API. Perhaps some kind of interface to allow the developer to choose how to store the data as it's parsed? There could then be several implementations, including a database one.

I created tests for each of the current data objects (nation, region, world, WA) and found some bugs (that are now fixed). The libraries I used didn't match (I thought they were the ones used in Android, but apparently the versions were slightly wrong), so I updated them so they are now tested to work. I also added the census data to the NationData object, so that one is now also complete. There is one unused shard in the incomplete WA object (lastresolution), as it returns invalid XML.

User avatar
Solm
Senator
 
Posts: 3582
Founded: Jul 23, 2008
Corporate Bordello

Postby Solm » Thu Feb 07, 2013 11:55 am

Afforess wrote:Similarly, keeping the xml in memory is also not ideal, with it being 5-30mb in size. Any other suggestions on a better way forward?


When I am doing any work with the dump, I made a custom parser that will read the file line by line and only save the bits I need in memory. So if I was parsing the dump for each nations flag, I only save the bits I need in a native format. Parsing the entire nations dump this way (115MB) takes about 20seconds. The regions dump takes around 5 or so seconds.

Retired: ns .hellodot. solm @ gmail .dot. com

User avatar
Afforess
Ambassador
 
Posts: 1105
Founded: Jun 22, 2009
Ex-Nation

Postby Afforess » Thu Feb 07, 2013 1:01 pm

Laevendell wrote:
Afforess wrote:Nice. I've only done some hacked up PHP that telegrams new nations that arrive to the Capitalist Paradise, not much else.

I've been considering doing something like that. Did you use a cron script to run it?


Yeah.

Laevendell wrote:
Afforess wrote:One of the other things I'm thinking about doing today is writing an OO representation of the regions and nations xml dumps, as they contain valuable information not in the shards. My only concern is that the sheer size of the xml data involved will make any sort of light-weight representation impossible - parsing the entire xml file once through, and then mapping the file to Java-based representations of the data is likely to produce thousands of objects. Similarly, keeping the xml in memory is also not ideal, with it being 5-30mb in size. Any other suggestions on a better way forward?

Well, to be easily accessible, but not take up memory, I'd probably have gone for a database, but I think we don't want to involve that in a light-weight API. Perhaps some kind of interface to allow the developer to choose how to store the data as it's parsed? There could then be several implementations, including a database one.


Hm, okay, In that case I will probably create a separate project for the dump representations. I think the database representation is the best plan.

Laevendell wrote:I created tests for each of the current data objects (nation, region, world, WA) and found some bugs (that are now fixed). The libraries I used didn't match (I thought they were the ones used in Android, but apparently the versions were slightly wrong), so I updated them so they are now tested to work. I also added the census data to the NationData object, so that one is now also complete. There is one unused shard in the incomplete WA object (lastresolution), as it returns invalid XML.


Great work - I've updated my repo with your changes.
Minister of the Interior, Capitalist Paradise

The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.

User avatar
Afforess
Ambassador
 
Posts: 1105
Founded: Jun 22, 2009
Ex-Nation

Postby Afforess » Thu Feb 07, 2013 4:15 pm

Solm wrote:
Afforess wrote:Similarly, keeping the xml in memory is also not ideal, with it being 5-30mb in size. Any other suggestions on a better way forward?


When I am doing any work with the dump, I made a custom parser that will read the file line by line and only save the bits I need in memory. So if I was parsing the dump for each nations flag, I only save the bits I need in a native format. Parsing the entire nations dump this way (115MB) takes about 20seconds. The regions dump takes around 5 or so seconds.

Right, a sax style XML parser makes the most sense. I've started a separate project, ns-dumps that fills a database with information from the regions.xml.gz data. It uses light-weight sax parsing to insert information for each region into the database. By default, it comes with an embedded h2 java database, but any SQL database could be used, it accepts configurable url connection strings, usernames/passwords.

I will try to finish the nations table/parsing in the next few days, it's more complicated than regions.

Project:
https://github.com/Afforess/ns-dumps
Minister of the Interior, Capitalist Paradise

The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.

User avatar
Afforess
Ambassador
 
Posts: 1105
Founded: Jun 22, 2009
Ex-Nation

Postby Afforess » Wed Apr 10, 2013 6:25 am

Been using the API for a while now, works really well. Hit an odd bug with this nations happenings though.

http://www.nationstates.net/cgi-bin/api ... happenings

In case the happenings update too quickly and are removed, I copied the XML below:

http://pastebin.com/3NkxhR9s

Seems that the HTML encoding is the culprit in this line:

Code: Select all
Following new legislation in @@ladyfasterkittens@@, Rupert Bear is considered to be the most risqué TV programme in LadyFasterkittens.


Don't have time to investigate further, will try to come up with a fix this evening. Just wanted to make you aware.
Minister of the Interior, Capitalist Paradise

The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.

User avatar
Ballotonia
Senior Admin
 
Posts: 5494
Founded: Antiquity
Democratic Socialists

Postby Ballotonia » Wed Apr 10, 2013 7:42 am

That's how the server provides the information: HTML is included in the tags without using CDATA escapes. Technically that's malformed XML, our fault. Sadly no easy fix other than dumping everything into CDATA tags, which might not make users happy either. We'd prefer to not have HTML there in the first place.

Note this gets way worse when a nation gets condemned or commended, in which case there's a blunt HTML link the happenings.

Ballotonia
"Een volk dat voor tirannen zwicht zal meer dan lijf en goed verliezen, dan dooft het licht…" -- H.M. van Randwijk

Next

Advertisement

Remove ads

Return to Technical

Who is online

Users browsing this forum: Astrobolt, Calegos

Advertisement

Remove ads