NationStates++ | Extension for NationStates

[Tlik]

Additionally, with the Real-Time RMB, could the timestamps update constantly? I'm going on the basis that because I've seen it on other websites it shouldn't be too hard, but I've no idea if it's possible in this case. It would be nice, though.

[[violet]]

Search function completed! Restart your browsers! You can search the entire RMB post history of any region.

Whoa! That is not good. The way you have done this, it will load the entire RMB, ten posts at a time, and load full nation data & flags for every single poster, regardless of whether their posts are contained in the search results.

For example, I searched for "gobbledygook" in Democratic Socialist Assembly and it issued 1,441 requests, of which about a thousand involved loading up to 10 nations.

I appreciate that there's a "Cancel" button, and the requests take place over a couple of minutes, but that is no good. One search in a medium-sized region can't require us to make 10,000 nation loads. Text-search of RMBs needs to happen server-side.

Also, I feel like the automatic 10-second refresh is too frequent. Someone walks away from their computer for half an hour and this extension will make 180 requests while they're gone.

[Afforess]

Search function completed! Restart your browsers! You can search the entire RMB post history of any region.

Whoa! That is not good. The way you have done this, it will load the entire RMB, ten posts at a time, and load full nation data & flags for every single poster, regardless of whether their posts are contained in the search results.

For example, I searched for "gobbledygook" in Democratic Socialist Assembly and it issued 1,441 requests, of which about a thousand involved loading up to 10 nations.

I appreciate that there's a "Cancel" button, and the requests take place over a couple of minutes, but that is no good. One search in a medium-sized region can't require us to make 10,000 nation loads. Text-search of RMBs needs to happen server-side.

Your search example is a worst case example. If you search for a common term like "the" it will only have to make 1-2 requests and then stop after 10 results are available. More requests are then only made if the user scrolls near the bottom of the page. Regarding the load, I have little understanding how the NationStates works in the backend, other than it's a system of flatfiles - but are you telling me that loading RMB posts also loads the entire nation information? I hesitate to criticize this wonderful site, but that seems like a really bad design.

I thought about storing RMB messages in a database of my own and having users see posts from that instead but the size requirements seem immense. Unless RMB text is a lot smaller than I suspect, I doubt I can afford to have 10gb+ databases on my tiny little server. Out of curiosity, how much disk space are RMB posts using on your end? Because the NationStates API does not allow Cross-Origin Requests I can not use that instead of the standard ajax query your site already makes.

I definitely don't want to be causing unreasonable bandwidth/server load on this site, so please make suggestions about how I could improve this.

Also, I feel like the automatic 10-second refresh is too frequent. Someone walks away from their computer for half an hour and this extension will make 180 requests while they're gone.

30 seconds seem more reasonable? I had it at 60 seconds earlier and it seemed too slow.

Additionally, with the Real-Time RMB, could the timestamps update constantly? I'm going on the basis that because I've seen it on other websites it shouldn't be too hard, but I've no idea if it's possible in this case. It would be nice, though.

Sure. I can add that.

[[violet]]

Your search example is a worst case example. If you search for a common term like "the" it will only have to make 1-2 requests and then stop after 10 results are available. More requests are then only made if the user scrolls near the bottom of the page.

Yep, I noticed that, but that's the best case. The worst case is a user in United North America searches for a word that doesn't exist; in that situation, the extension sends us about 12,000 requests.

Regarding the load, I have little understanding how the NationStates works in the backend, other than it's a system of flatfiles - but are you telling me that loading RMB posts also loads the entire nation information? I hesitate to criticize this wonderful site, but that seems like a really bad design.

Looking up almost anything about a nation requires the whole thing to be loaded. it's not possible to just extract its flag or type or whatever. Which is indeed a really bad design, for this kind of thing. But it's the way we are.

I definitely don't want to be causing unreasonable bandwidth/server load on this site, so please make suggestions about how I could improve this.

The RMB database is about 500M and it's no problem to search it. For example, if I simulate the "gobbledygook" search via SQL, I get an "Empty set" response in 0.03 seconds. Which is why I say text search needs to happen server-side: that's 0.03 seconds and 0 nation loads vs the extension's 1,441 requests and 10,000 nation loads. There's nothing you can do to improve its searching ability short of what you've already done (e.g. stopping after X successful hits), because you can't search the whole RMB without fetching it all first.

So what we need to do is let your extension access those search results, so it can make a request like "/page=ajax/a=rmbsearch/region=capitalist_paradise/text=Hello/start=0/limit=10". Then it can display the result.

Of course, then I would probably steal the feature for the rest of the site. And then you would code some more stuff, and I would get sick of playing catch-up, and you could become an admin and do it yourself. But for now, this would work.

Also, I feel like the automatic 10-second refresh is too frequent. Someone walks away from their computer for half an hour and this extension will make 180 requests while they're gone.

30 seconds seem more reasonable? I had it at 60 seconds earlier and it seemed too slow.

Yeah, or if you're feeling ambitious, have it fetch every 10s for the first 2 minutes, then steadily less frequently, on the assumption that if the user hasn't done anything for 5 minutes, they're probably not sitting there just staring at the page. I don't mind fairly frequent requests if they're actually useful; it's the pointless requests that I want squashed.

[Afforess]

Your search example is a worst case example. If you search for a common term like "the" it will only have to make 1-2 requests and then stop after 10 results are available. More requests are then only made if the user scrolls near the bottom of the page.

Yep, I noticed that, but that's the best case. The worst case is a user in United North America searches for a word that doesn't exist; in that situation, the extension sends us about 12,000 requests.

Regarding the load, I have little understanding how the NationStates works in the backend, other than it's a system of flatfiles - but are you telling me that loading RMB posts also loads the entire nation information? I hesitate to criticize this wonderful site, but that seems like a really bad design.

Looking up almost anything about a nation requires the whole thing to be loaded. it's not possible to just extract its flag or type or whatever. Which is indeed a really bad design, for this kind of thing. But it's the way we are.

That's unfortunate. If I start keeping track of this sort of data myself, will it still be a problem? If I have a direct link to a nations flag for example, without hitting ns.net or the api, will that cause a nation load?

I definitely don't want to be causing unreasonable bandwidth/server load on this site, so please make suggestions about how I could improve this.

The RMB database is about 500M and it's no problem to search it. For example, if I simulate the "gobbledygook" search via SQL, I get an "Empty set" response in 0.03 seconds. Which is why I say text search needs to happen server-side: that's 0.03 seconds and 0 nation loads vs the extension's 1,441 requests and 10,000 nation loads. There's nothing you can do to improve its searching ability short of what you've already done (e.g. stopping after X successful hits), because you can't search the whole RMB without fetching it all first.

So what we need to do is let your extension access those search results, so it can make a request like "/page=ajax/a=rmbsearch/region=capitalist_paradise/text=Hello/start=0/limit=10". Then it can display the result.

Wait - so the RMB posts are actually in a separate database? If so - way cool. If it's easy to expose a query like that, I would switch over in a heartbeat.

Of course, then I would probably steal the feature for the rest of the site. And then you would code some more stuff, and I would get sick of playing catch-up, and you could become an admin and do it yourself. But for now, this would work.

Where do I sign up for this alternate reality?

30 seconds seem more reasonable? I had it at 60 seconds earlier and it seemed too slow.

Yeah, or if you're feeling ambitious, have it fetch every 10s for the first 2 minutes, then steadily less frequently, on the assumption that if the user hasn't done anything for 5 minutes, they're probably not sitting there just staring at the page. I don't mind fairly frequent requests if they're actually useful; it's the pointless requests that I want squashed.

Thats... a really good idea. I'll implement it that way!

[[violet]]

Looking up almost anything about a nation requires the whole thing to be loaded. it's not possible to just extract its flag or type or whatever. Which is indeed a really bad design, for this kind of thing. But it's the way we are.

That's unfortunate. If I start keeping track of this sort of data myself, will it still be a problem? If I have a direct link to a nations flag for example, without hitting ns.net or the api, will that cause a nation load?

No, it wouldn't. But you probably shouldn't bother trying to optimize like that except in extreme situations, since you can't see what's happening on our side. E.g. it's worth avoiding loading nations thousands of times when you don't need them at all, but it's not worth caching flags for eight nations when you're displaying ten search results (and we'll probably load them anyway when we fetch their capitalized name).

So what we need to do is let your extension access those search results, so it can make a request like "/page=ajax/a=rmbsearch/region=capitalist_paradise/text=Hello/start=0/limit=10". Then it can display the result.

Wait - so the RMB posts are actually in a separate database? If so - way cool. If it's easy to expose a query like that, I would switch over in a heartbeat.

It is easy. We maybe want to think about what's next, though. After this, will you want to add searching for RMB posts by a particular nation? Or by date? What about combining search criteria? It would be easier to build the interface once properly than reconfigure it all the time for new features.

Of course, then I would probably steal the feature for the rest of the site. And then you would code some more stuff, and I would get sick of playing catch-up, and you could become an admin and do it yourself. But for now, this would work.

Where do I sign up for this alternate reality?

We are always looking to sucker in appoint new admins. It's a long slow process because of security reasons, i.e. it would be bad if we gave someone code access and they did something naughty with it. But if you're interested in writing code for the game then I'm interested in using it.

You could also do something like Northrop-Grumman's "Dark" theme, which began as a browser extension and is now available to all players directly from the Settings page. N-G builds that by himself and emails in patches. You could have a theme that supplies its own JavaScript.

[Afforess]

Wait - so the RMB posts are actually in a separate database? If so - way cool. If it's easy to expose a query like that, I would switch over in a heartbeat.

It is easy. We maybe want to think about what's next, though. After this, will you want to add searching for RMB posts by a particular nation? Or by date? What about combining search criteria? It would be easier to build the interface once properly than reconfigure it all the time for new features.

True. I was thinking of designing an advanced search form. I agree that designing the interface first - then implementing is the right approach. The obvious ones are:

Keywords, with AND, OR options
By poster
By nations mentioned (inside [nation] tags)
By regions mentioned (inside [region] tags)
Older than + Newer than options
...
others that I am forgetting?

Where do I sign up for this alternate reality?

We are always looking to sucker in appoint new admins. It's a long slow process because of security reasons, i.e. it would be bad if we gave someone code access and they did something naughty with it. But if you're interested in writing code for the game then I'm interested in using it.

You could also do something like Northrop-Grumman's "Dark" theme, which began as a browser extension and is now available to all players directly from the Settings page. N-G builds that by himself and emails in patches. You could have a theme that supplies its own JavaScript.

Definitely interested, just never asked because there is an unspoken rule that the people who ask for moderator/admin powers are the people who least deserve them. If you are serious though,I am interested and you should have my email and can contact me about starting the process that way. If not, well I will just keep writing browser extensions!

[Afforess]

This is awesome. Great work.

EDIT:

- Real time RMB: Loads new RMB posts in the page, without refreshing

Do you think you could apply this to telegrams as well?

I can make the "Telegrams" indicator on the sidebar update the count, but I don't want to add features to the actual telegrams page (infinite scroll/search). I wouldn't trust a browser extension to read telegrams, and neither should you!

[Feux]

This is awesome. Great work.

EDIT:


Do you think you could apply this to telegrams as well?

I can make the "Telegrams" indicator on the sidebar update the count, but I don't want to add features to the actual telegrams page (infinite scroll/search). I wouldn't trust a browser extension to read telegrams, and neither should you!

I'm talking about the telegrams indicator updating.

[Afforess]

I can make the "Telegrams" indicator on the sidebar update the count, but I don't want to add features to the actual telegrams page (infinite scroll/search). I wouldn't trust a browser extension to read telegrams, and neither should you!

I'm talking about the telegrams indicator updating.

Telegram indicator updating and RMB post time updating should both be live. Clear your cache if you don't see it working.

[[violet]]

I've added a search backend for RMBs. Currently it can only be accessed via calls to /page=ajax/a=rmbsearch. Then you throw in extra fields like:

• rmbsearch-text=(text to search for)
• rmbsearch-author=(real_name of nation to find posts from)
• rmbsearch-region=(real_name of region to limit search to)
• rmbsearch-limit=(max. number of posts to retrieve, default is 20, cannot exceed 20)
• rmbsearch-offset=(skip first X number of search results)
• rmbsearch-reverse=(1 if you want results in reverse order)

All of the above may change and is just for playing around with to help us figure out what a good RMB search interface should look like.

Edit: Oh and the search text works like this, so you can use tokens to search for phrases, exclude words, etc.

Edit 2: Example: Who's mentioned Testlandia in Osiris?

[Afforess]

I've added a search backend for RMBs. Currently it can only be accessed via calls to /page=ajax/a=rmbsearch. Then you throw in extra fields like:

• rmbsearch-text=(text to search for)
• rmbsearch-author=(real_name of nation to find posts from)
• rmbsearch-region=(real_name of region to limit search to)
• rmbsearch-limit=(max. number of posts to retrieve, default is 20, cannot exceed 20)
• rmbsearch-offset=(skip first X number of search results)
• rmbsearch-reverse=(1 if you want results in reverse order)

All of the above may change and is just for playing around with to help us figure out what a good RMB search interface should look like.

Edit: Oh and the search text works like this, so you can use tokens to search for phrases, exclude words, etc.

Edit 2: Example: Who's mentioned Testlandia in Osiris?

Excellent! I'll switch over to it very soon.

[Afforess]

@Violet
What is the correct way to search for text with spaces in it? The html encoding changes spaces to %20 and that seems to be ignored.

Also while I am getting my wishes granted, some sort of ajax for 'Today's World Census Report' would be nice. Opening pages right now forces an entire load of the region page. I added a slider earlier today (before I you added the text searching) and have it delay 250ms before loading (so sliding back and forth wildly does not trigger tons of requests), but having an ajax for just the table data obviously would be best.

[[violet]]

The html encoding changes spaces to %20 and that seems to be ignored.

I think maybe what you're seeing is the implied OR between words. Which seems weird to me, but that's the MySQL implementation. You need a "+" before the word to make it an implied AND. Which would be encoded in an URI as %2B.

[Afforess]

The html encoding changes spaces to %20 and that seems to be ignored.

I think maybe what you're seeing is the implied OR between words. Which seems weird to me, but that's the MySQL implementation. You need a "+" before the word to make it an implied AND. Which would be encoded in an URI as %2B.

Yep, you're right. Works fine.

[TaQud]

@Afforness:

Question: A post I made was now merged into a topic that I don't want to see. Can you somehow make a script or add-on that let's you ignore or block a topic from your "view your posts"?

[Afforess]

I've added a search backend for RMBs. Currently it can only be accessed via calls to /page=ajax/a=rmbsearch. Then you throw in extra fields like:

• rmbsearch-text=(text to search for)
• rmbsearch-author=(real_name of nation to find posts from)
• rmbsearch-region=(real_name of region to limit search to)
• rmbsearch-limit=(max. number of posts to retrieve, default is 20, cannot exceed 20)
• rmbsearch-offset=(skip first X number of search results)
• rmbsearch-reverse=(1 if you want results in reverse order)

All of the above may change and is just for playing around with to help us figure out what a good RMB search interface should look like.

Edit: Oh and the search text works like this, so you can use tokens to search for phrases, exclude words, etc.

Edit 2: Example: Who's mentioned Testlandia in Osiris?

Alright, I have the new text search mostly working but there seem to be some idiosyncrasies in how it works. It seems to hate '-' characters:

http://www.nationstates.net/page=ajax/a ... t_paradise

No Messages? I doubt no one has ever mentioned the founder of the region.

Also there is apparently a 4 letter minimum limit for any word you are searching for, which seems sensible, except it even applies to quoted text, like '"You are a"'. Quoted, it makes up a 9 letter phrase to search for, but it returns no results.

http://www.nationstates.net/page=ajax/a ... t_paradise

[[violet]]

These are quirks of the MySQL search engine--see the page I linked earlier. For example, "-" is interpreted as "NOT", so "The-CID" is a search for posts containing the word "The" but not "CID". You can avoid that by searching for it as a phrase, i.e. enclosed in quotation marks.

More fundamentally, though, the search engine indexes only whole words, ignoring punctuation, and excludes really common ones, so searching for "The <anything>" doesn't work. It just ignores the "The". We would have to lower the minimum word length and remove "the" from the list of stopwords, which might be impractical. (It will search for all posts containing the word "the", then try to cross-reference them against posts containing "CID".)

[New Worcester]

Violet? Are you Max Barry? It seems that you don't any nation.

[Afforess]

Violet? Are you Max Barry? It seems that you don't any nation.

Violet is definitely not Max Barry, and does not need a nation as she is an administrator. Please stay on topic.

[Corporation de Apple]

This is an UTTERLY fantastic tool - thanks a ton.

One cosmetic thing, though.
I hate non-continuity, and I noticed that you're using Bootstrap buttons for the "Move to" and "Add to Dossier", which looks great.
However, the RMB buttons are a smidgen different.
Perhaps do somefink like this:

Code: Select all

<div class="btn-group">
  <button class="btn btn-medium">Leave a Message</button>
  <button class="btn btn-medium">Search Messages</button>
</div>


And for quote:

Code: Select all

<span class="badge">Quote</span>

I'm at work right now, so I can't access GitHub and check your code, but that might prettify the quotes/leave message thing even more.
Just a thought, and thanks a bajillion.

[Astarial]

This is very, very cool.

I've noticed one thing that's potentially irksome - If I search for a term, there's no way as far as I can tell to switch back to the main RMB view. I have to click "leave a message" twice for that. When I click the search button again the search box goes away - would it be a relatively simple fix to make that also reload the RMB proper?

Other than that, haven't noticed any problems so far.

[Crushing Our Enemies]

First off, this whole think is awesome. You are doing amazing work for NationStates players.

That being said, I don't have a use for most of these features, and kind of dis-prefer some of the cosmetic changes this add-on makes. However, I LOVE some of these features, and would like be able to pick and choose which to turn on and off. Any chance that users could someday get the capability to toggle each feature individually?

[Hileville]

First off, this whole think is awesome. You are doing amazing work for NationStates players.

That being said, I don't have a use for most of these features, and kind of dis-prefer some of the cosmetic changes this add-on makes. However, I LOVE some of these features, and would like be able to pick and choose which to turn on and off. Any chance that users could someday get the capability to toggle each feature individually?

I completely agree wit this. I would prefer an option to disable a couple of the features and would switch to this immediately.

[Afforess]

First off, this whole think is awesome. You are doing amazing work for NationStates players.

That being said, I don't have a use for most of these features, and kind of dis-prefer some of the cosmetic changes this add-on makes. However, I LOVE some of these features, and would like be able to pick and choose which to turn on and off. Any chance that users could someday get the capability to toggle each feature individually?

First off, this whole think is awesome. You are doing amazing work for NationStates players.

That being said, I don't have a use for most of these features, and kind of dis-prefer some of the cosmetic changes this add-on makes. However, I LOVE some of these features, and would like be able to pick and choose which to turn on and off. Any chance that users could someday get the capability to toggle each feature individually?

I completely agree wit this. I would prefer an option to disable a couple of the features and would switch to this immediately.

I've been planning on adding a settings area so you can enable/disable features. What exactly are you looking to be able to turn off?