RMB search & general server performance

[[violet]]

Just a note that I've had to disable RMB search for the time being, since it was leading to a cascade of performance issues.

Things have been pretty bogged down for the last two weeks, but hopefully this improves things somewhat.

[[violet]]

A few more optimizations:

Fixed a problem related to The Embassy which had become too large to fit within the in-memory cache, thus requiring it to be painstakingly loaded from disk every time. This mainly affected pages that mentioned The Embassy in their Happenings, creating a 5-15s slowdown.

Added a workaround for a problem relating to very large RMBs, where it is quite slow to calculate the pagination of a post when looking it up via its post ID with no other information. This is still slow, but the server doesn't need to figure it out as often. In particular, this should be noticeable when posting a new message and then viewing your own post.

Reduced the amount of logging we do, which has a cascading effect on performance when things get slow.

[Flanderlion]

Just a note that I've had to disable RMB search for the time being, since it was leading to a cascade of performance issues.

Things have been pretty bogged down for the last two weeks, but hopefully this improves things somewhat.

Just saw RMB search was disabled - what's changed that caused this to start being a problem? Not particularly concerned about it being disabled, just more curious as to why it started becoming an issue.

[[violet]]

Growth of RMBs has meant that the database's inbuilt search just isn't fast enough any more.

[Great Algerstonia]

Growth of RMBs has meant that the database's inbuilt search just isn't fast enough any more.

Would it possibly be reintroduced in the future? And is this a threat with forum searches as well?

[9003]

Could it be reasonable to archive old rbm posts? Even just lopping off some of the ones from 5+years and putting them in a zip for the region rather than haveing the search function have to dig thorugh all of them?

[Twilight Sparkle]

Oh dear, this has some detrimental effects on regions with active RMB roleplaying. I hope this will be fixed soon.

[[violet]]

Growth of RMBs has meant that the database's inbuilt search just isn't fast enough any more.

Would it possibly be reintroduced in the future? And is this a threat with forum searches as well?

The forum has a dedicated search service, so it's fine. To fix the RMB situation, we need to add one of those.

[Lenlyvit]

Would it possibly be reintroduced in the future? And is this a threat with forum searches as well?

The forum has a dedicated search service, so it's fine. To fix the RMB situation, we need to add one of those.

Possibly a stupid question, but are you working on one in your available time? Also, thanks for finding the problems and actively working to solve them. I know it's a tiresome job, but I just want to remind you admin that if it weren't for you NS would've died long ago and we really appreciate having you around

[[violet]]

Thanks! Yep, I have spent time on it and hope to implement soon.

[Twilight Sparkle]

Thanks! Yep, I have spent time on it and hope to implement soon.

https://derpibooru.org/62924

[Flanderlion]

Thanks! Yep, I have spent time on it and hope to implement soon.

We love you [v]

[The Ambassadors Reception]

A few more optimizations:

Fixed a problem related to The Embassy which had become too large to fit within the in-memory cache, thus requiring it to be painstakingly loaded from disk every time. This mainly affected pages that mentioned The Embassy in their Happenings, creating a 5-15s slowdown.

Added a workaround for a problem relating to very large RMBs, where it is quite slow to calculate the pagination of a post when looking it up via its post ID with no other information. This is still slow, but the server doesn't need to figure it out as often. In particular, this should be noticeable when posting a new message and then viewing your own post.

Reduced the amount of logging we do, which has a cascading effect on performance when things get slow.

I've just spotted this thread. Thanks so much for this enhancement. It has made an enormous difference to my life! That was a painful few weeks of waiting for Embassy pages to load. I look forward to the search update too. You're a star!

[[violet]]

RMB Search is now reimplemented.

[Kuriko]

RMB Search is now reimplemented.

Thanks [v]!!! You're the best!!!!

- Kuriko
Aka Lenlyvit

[Flanderlion]

RMB Search is now reimplemented.

3 char etc. searches. Not just reimplemented, massively improved. Thank you so much.

[Jar Wattinree]

RMB Search is now reimplemented.

You are amazing.

[Phydios]

RMB Search is now reimplemented.

Thank you for being so quick- and improving the search function at the same time! There's years of discussion on our RMB, and I feared that finding old conversations would no longer be easy.

[Trotterdam]

I notice there's been a change in how multiple-word searches work.

Previously, a search for "aaaa bbbb cccc" would use an implied logical OR, so it would find posts that contain any one of those three words. To find only posts that include all three, you needed to type "+aaaa +bbbb +cccc".

Now, it appears to default to a logical AND, so "aaaa bbbb cccc" finds only posts that include all three words. I found through simple experimentation with formats used by other search engines that "OR" is simply treated as a search term rather than a logical operator, but "|" does work, so "aaaa | bbbb | cccc" finds all posts that include at least one of the words (in 0.00 seconds! yay).

I don't mind this, but it would be nice to have some documentation on which operators are implemented in the RMB search.

[Aclion]

RMB Search is now reimplemented.

\o/

[[violet]]

I don't mind this, but it would be nice to have some documentation on which operators are implemented in the RMB search.

I haven't quite figured it out myself, but it should be either this or this. Maybe you could write up a nice dispatch!

[Trotterdam]

I don't mind this, but it would be nice to have some documentation on which operators are implemented in the RMB search.

I haven't quite figured it out myself, but it should be either this or this. Maybe you could write up a nice dispatch!

Looks like the latter. Almost everything on there seems to work, including some of the more obscure operators like ^, $, <<, ""~, and NEAR/, as well as the more important |, -, "", and ().

Although I don't understand the purpose of the = operator, since everything already seems to do exact matches anyway (a search for "cat" did not find a post with only "cats", a search for "Pokemon" did not find a post with only "Pokémon", or to use the manual's own example, a search for "runs" did not find a post with only "running" - I'm guessing that feature only works with a dictionary that NationStates doesn't have), or the MAYBE operator, since searching for x MAYBE y seems to do the same as just searching for x (maybe it affects priority when sorting by relevance?). Also the field stuff would be inapplicable since we don't actually have multiple fields, except the author field, which is treated separately on the search form.

SENTENCE and PARAGRAPH also don't seem to quite work reliably, even though they are recognized. I suppose there's not much call for those, though.

One feature we seem to have lost is partial word matches, as in trot* matching Trotterdam. Combined with only exact matches being found, that means you'd often need to type stuff like cat | cats if you're not sure of the exact form a word will be in.

Another minor nitpick is that entering an invalid search query (for example, -invalid, since a query must contain at least one positive term) gives "System Error (eval) at <date/time>." rather than a more informative error message.

Aside from that, everything works great! Thanks for implementing this.

[[violet]]

Aha, thanks. I've created an RMB Search Help dispatch.

Although I don't understand the purpose of the = operator, since everything already seems to do exact matches anyway

Ah yes... that's probably for setups that have morphology enabled, which we don't. The search engine can attempt to reduce words down to stems and match on the root, but in my testing it often returned a lot of unrelated results, I think because we have a database full of invented words.

[Nakarisaune]

When searching for something with more than 1000 results, the pagination only allows me to get to page 50 (981–1000) and there doesn't seem to be a way to find posts further back. Is this a bug or a limitation of the search system?

[[violet]]

When searching for something with more than 1000 results, the pagination only allows me to get to page 50 (981–1000) and there doesn't seem to be a way to find posts further back. Is this a bug or a limitation of the search system?

The latter. I think the idea is that if you're getting that many hits, you should narrow your search terms, rather than start paging through thousands of posts.