NationStates++ | Extension for NationStates

[Tlik]

I liekz the happenings feed.

It gets really crazy to watch around the update times. Thousands of happenings.

How are you doing that, by the way? How often do you check each nation page for the happenings?

[Afforess]

It gets really crazy to watch around the update times. Thousands of happenings.

How are you doing that, by the way? How often do you check each nation page for the happenings?

It's pretty ingenious - it doesn't use the NS API, it scrapes the site. A players dossier can hold 2500 nations at once. Your reports page shows the happenings for all of those nations. I can clear the dossier with 1 request and use the advanced upload post form to cycle the next 2500 nations, rinse repeat. So i can upload 2500 nations to the dossier, scrap the reports page, then clear the dossier in 3 requests. I can collect happenings on all of NS in 43 batches and under 15 min.

The one downside is that I don't get exact time stamps, I have to parse and make a best guess. In fact the reports pages are so large it takes several seconds to parse and update happenings on my tiny server. Good thing the HTML site rate limit is roughly the same length of time.

I have 1.6M happenings stored already, adding ~70k a day. I have plans to begin to open up and document my backend server so other people can use it. I already use CORS/JSON for everything.

I think it's a pretty clever solution if I so say so myself.

[Tlik]

How are you doing that, by the way? How often do you check each nation page for the happenings?

It's pretty ingenious - it doesn't use the NS API, it scrapes the site. A players dossier can hold 2500 nations at once. Your reports page shows the happenings for all of those nations. I can clear the dossier with 1 request and use the advanced upload post form to cycle the next 2500 nations, rinse repeat. So i can upload 2500 nations to the dossier, scrap the reports page, then clear the dossier in 3 requests. I can collect happenings on all of NS in 43 batches and under 15 min.

The one downside is that I don't get exact time stamps, I have to parse and make a best guess. In fact the reports pages are so large it takes several seconds to parse and update happenings on my tiny server. Good thing the HTML site rate limit is roughly the same length of time.

I have 1.6M happenings stored already, adding ~70k a day. I have plans to begin to open up and document my backend server so other people can use it. I already use CORS/JSON for everything.

I think it's a pretty clever solution if I so say so myself.

Have you thought about using multiple nations, each with a portion of the total dossier? It would be harder to programatically ensure that each nation exists and add new nations, but it might save on the upload parts. I'm not sure how much of a saving you'd get, though, it depends how differently NS 'feels' logging into a new nation vs uploading a new dossier.

[TaQud]

i like the gameplay news

[Corporation de Apple]

i like the gameplay news

*Gamplay

Tyops, there are everywhere. Apart from that, it looks awesome.
In nationstates.js, Line 8 (per Notepad++)

Code: Select all

$("<li id='ns_newspaper'><a id='ns_newspaper_link' href='http://www.nationstates.net/page=news/?ns_newspaper=true'>GAMPLAY NEWS</a></li>").insertAfter($("#live_happenings_feed"));

Becooomes...

Code: Select all

$("<li id='ns_newspaper'><a id='ns_newspaper_link' href='http://www.nationstates.net/page=news/?ns_newspaper=true'>GAMEPLAY NEWS</a></li>").insertAfter($("#live_happenings_feed"));

[Usual People In Life]

Looks nice, I'll download the Chrome version next time I use my laptop!

I don't suppose you could make a Nintendo 3DS flavour of this extension (the browser uses the NetFront engine)?

[The Cosmos]

Gameplay News is awesome! However, I noticed that you misspelled Gameplay as "Gamplay"

[Afforess]

It's pretty ingenious - it doesn't use the NS API, it scrapes the site. A players dossier can hold 2500 nations at once. Your reports page shows the happenings for all of those nations. I can clear the dossier with 1 request and use the advanced upload post form to cycle the next 2500 nations, rinse repeat. So i can upload 2500 nations to the dossier, scrap the reports page, then clear the dossier in 3 requests. I can collect happenings on all of NS in 43 batches and under 15 min.

The one downside is that I don't get exact time stamps, I have to parse and make a best guess. In fact the reports pages are so large it takes several seconds to parse and update happenings on my tiny server. Good thing the HTML site rate limit is roughly the same length of time.

I have 1.6M happenings stored already, adding ~70k a day. I have plans to begin to open up and document my backend server so other people can use it. I already use CORS/JSON for everything.

I think it's a pretty clever solution if I so say so myself.

Have you thought about using multiple nations, each with a portion of the total dossier? It would be harder to programatically ensure that each nation exists and add new nations, but it might save on the upload parts. I'm not sure how much of a saving you'd get, though, it depends how differently NS 'feels' logging into a new nation vs uploading a new dossier.

It wouldn't be worth the extra complexity, considering that I would have to still manage the 43 puppets and keep their dossiers up to date (with regards to new nations and dead ones).

Gameplay News is awesome! However, I noticed that you misspelled Gameplay as "Gamplay"

Oops. It was 3am and I was tired.

[The Republic of Lanos]

Is there a way to disable the gameplay and happening feeds?

[Afforess]

Is there a way to disable the gameplay and happening feeds?

There is now. Check your settings.

[Libetarian Republics]

You sir are amazing.

[Glen-Rhodes]

Please check your changes with the Antiquity theme. The Happenings Feed link is going crazy with that exclamation point.

[[violet]]

How are you doing that, by the way? How often do you check each nation page for the happenings?

It's pretty ingenious - it doesn't use the NS API, it scrapes the site. A players dossier can hold 2500 nations at once. Your reports page shows the happenings for all of those nations. I can clear the dossier with 1 request and use the advanced upload post form to cycle the next 2500 nations, rinse repeat. So i can upload 2500 nations to the dossier, scrap the reports page, then clear the dossier in 3 requests. I can collect happenings on all of NS in 43 batches and under 15 min.

Unfortunately, this is ridiculously load-generating for us. The only reason dossiers can hold 2500 nations is because few people ever exercise that limit. We can't actually support generating real-time reports on 2,500 nations on a regular basis. (Which is why it's not part of the API.)

I don't want to lower everyone's dossier limit, but I will have to do that or something like it if it's used in this way.

Edit: Oh, is this just you, scraping to a central server? I was thinking every NS++ user would be doing it. That's not so awful.

[[violet]]

LOVE Gameplay News, by the way, and Live Happenings are super-interesting to watch.

[Afforess]

It's pretty ingenious - it doesn't use the NS API, it scrapes the site. A players dossier can hold 2500 nations at once. Your reports page shows the happenings for all of those nations. I can clear the dossier with 1 request and use the advanced upload post form to cycle the next 2500 nations, rinse repeat. So i can upload 2500 nations to the dossier, scrap the reports page, then clear the dossier in 3 requests. I can collect happenings on all of NS in 43 batches and under 15 min.

Unfortunately, this is ridiculously load-generating for us. The only reason dossiers can hold 2500 nations is because few people ever exercise that limit. We can't actually support generating real-time reports on 2,500 nations on a regular basis. (Which is why it's not part of the API.)

I don't want to lower everyone's dossier limit, but I will have to do that or something like it if it's used in this way.

Edit: Oh, is this just you, scraping to a central server? I was thinking every NS++ user would be doing it. That's not so awful.

It's just one central server. I store all the happenings in a database, and then fetch them from there. All NS++ users query my server and retrieve the database records, so only 1 account is scraping, it's Capitalist Paradise Free Press. If it becomes a problem, I imagine you could simply lock out that account, which would stop the scraping. I also am trying to observe the regular site rate limits too - so it shouldn't be occurring too frequently.

You know what would be awesome? If all happenings in nationstates included the exact timestamp in a <span> element attribute. Right now I have to do guesswork, but I know NS has the exact times. You could simple change the "3 minutes ago: bla bla bla" into "<span time='_some_exact_timestamp'>3 minutes ago</span>: bla bla bla". Users wouldn't see any difference, but it would mean I could stop polling the NS region/nation/world happenings every 30s for timestamp updates. I am already doing this for the live happenings feed - if you check out the html I generate, it has the timestamp in the html, and then I can use that to update the html description as needed.

[Afforess]

LOVE Gameplay News, by the way, and Live Happenings are super-interesting to watch.

I honestly threw threw the gameplay news together in under 20 minutes. I plan on revisiting it and improving the design. It's just an iframe with a google doc link.

[Afforess]

i like the gameplay news

*Gamplay

Tyops, there are everywhere. Apart from that, it looks awesome.
In nationstates.js, Line 8 (per Notepad++)

Code: Select all

$("<li id='ns_newspaper'><a id='ns_newspaper_link' href='http://www.nationstates.net/page=news/?ns_newspaper=true'>GAMPLAY NEWS</a></li>").insertAfter($("#live_happenings_feed"));

Becooomes...

Code: Select all

$("<li id='ns_newspaper'><a id='ns_newspaper_link' href='http://www.nationstates.net/page=news/?ns_newspaper=true'>GAMEPLAY NEWS</a></li>").insertAfter($("#live_happenings_feed"));

I'm amazed people honestly check out the code. Sometimes I think I am the only one who sees it.

Please check your changes with the Antiquity theme. The Happenings Feed link is going crazy with that exclamation point.

The Aniquity theme is the bane of my existence. It does everything in a slightly different and confusing to handle way from the default & dark theme. Thanks for the report... and don't think I am chastising you... do keep reporting bugs. I'll go patch this one up.

Edit: Oh if it wasn't clear, the "!" is supposed to occur when a very large amount of happenings are being created, generally during major and minor game updates. I can't imagine a situation outside of those that would generate enough happenings in a short enough timespan to trigger it. If you have no idea what I am talking about, watch the sidebar around an update.

[Avalon]

Man, I love this app, this thread and this community. Admins trying to help, users coming up with ideas. This is what NS is all about after all.

Thanks again Afforess and all

[[violet]]

Okay, I'm doing it: I will make real-time Happenings available. It's just too embarrassing to watch you have to jump through these hoops to make something cool.

So what we should have sometime next week is something with proper timestamps that you can query as often as you like.

[Glen-Rhodes]

The Aniquity theme is the bane of my existence. It does everything in a slightly different and confusing to handle way from the default & dark theme. Thanks for the report... and don't think I am chastising you... do keep reporting bugs. I'll go patch this one up.

I know how you feel. When I was still doing this kind of stuff, it was always a pain to make special changes for the Antiquity theme. But I refuse to join the rest of NS in using these newfangled themes. Unfortunately, I have to point out that the Jennifer Government logo is right in the middle of the Gameplay News page. :|

[Afforess]

Okay, I'm doing it: I will make real-time Happenings available. It's just too embarrassing to watch you have to jump through these hoops to make something cool.

So what we should have sometime next week is something with proper timestamps that you can query as often as you like.

Yesss!

[Afforess]

The Aniquity theme is the bane of my existence. It does everything in a slightly different and confusing to handle way from the default & dark theme. Thanks for the report... and don't think I am chastising you... do keep reporting bugs. I'll go patch this one up.

I know how you feel. When I was still doing this kind of stuff, it was always a pain to make special changes for the Antiquity theme. But I refuse to join the rest of NS in using these newfangled themes. Unfortunately, I have to point out that the Jennifer Government logo is right in the middle of the Gameplay News page. :|

Fixed.

[[violet]]

I swear Antiquity adds 10% to the development time of every new feature. It's a pain in the butt.

But it is nice to be able to view the site in its original form.

[Afforess]

I swear Antiquity adds 10% to the development time of every new feature. It's a pain in the butt.

But it is nice to be able to view the site in its original form.

Speaking of bugs, an admin should check out my dossier. I somehow have duplicate nations in it. I'll not clear it so you can check it out.

[Nightkill the Emperor]

Is there any way to allow us to see what subforum we've posted most in (General, II, P2TM) and what topic?

I remember that was briefly a feature we had in the game, but was gone for some reason after a week.