Scripts banned

[[violet]]

I just had to ban two script IPs that were hitting nationdata.cgi at an unacceptably high rate (multiple times per second). One identified itself as NSEconomy and the other did not identify itself. Please let me know if you own these scripts and are now getting "403 Forbidden" errors.

[East Fancainia]

I just had to ban two script IPs that were hitting nationdata.cgi at an unacceptably high rate (multiple times per second). One identified itself as NSEconomy and the other did not identify itself. Please let me know if you own these scripts and are now getting "403 Forbidden" errors.

Why would NSE be doing that? Could this be the cause of my computer running slow on Nationstates for like the past minute?

[Collective Systems]

I was getting that error a few minutes ago, but not anymore. This website still seems a tad slow, however.

[Unidox]

I just had to ban two script IPs that were hitting nationdata.cgi at an unacceptably high rate (multiple times per second). One identified itself as NSEconomy and the other did not identify itself. Please let me know if you own these scripts and are now getting "403 Forbidden" errors.

... Could this be the cause of my computer running slow on Nationstates for like the past minute?

I've noticed this too. Although, forum lag time usually has something to do with the server, or the number of Ad Bots on today.

[Frenca]

I just had to ban two script IPs that were hitting nationdata.cgi at an unacceptably high rate (multiple times per second). One identified itself as NSEconomy and the other did not identify itself. Please let me know if you own these scripts and are now getting "403 Forbidden" errors.

That's probably the reason why my computer was acting weird here...

[[violet]]

I'm not sure exactly how NSE works, but possibly it waits until someone types in a region name, then hits us up for data on every nation in that region very quickly. And possibly it's not storing that data, either, so if one person looks up The Pacific three times in a minute, NSEconomy tries to load every Pacific nation three times in quick succession.

If that's the case, NSE needs to be smarter about how it gets data from us. It's fine to trawl the site slowly, but hitting us with a flood of requests all of a sudden ties up the server, slowing it for everyone else.

We were running pretty slow but seem to be back to normal now.

[Schultaria Prime]

Don't know why, but when I logged on to NSEconomy I received economic data for over 15,000 nations (including one that I let die a couple of weeks ago) instead of the usual welcome screen. As far as I can recollect NSEconomy should have built in brakes from accessing economic data from feeder regions and other large entities, but it seems like something is quite wrong with the website at the moment.

(CORRECTION) It's when I access NSEconomy through a search engine that it seems to do this. Otherwise, it loads up normally.

[Collective Systems]

Na, it's still a bit slow for me.

I don't understand why you allow bots unto this website. Can't you just put one of those picture-code-a-ma-bobs when creating a new nation?

[Redneck United States]

im gonna go out on a limb here and claim its the Liberal NationStates thing...as the second unidentified

[Collective Systems]

im gonna go out on a limb here and claim its the Liberal NationStates thing...as the second unidentified

"Aprilway Oolsfay Ayday."

[[violet]]

Na, it's still a bit slow for me.

I don't understand why you allow bots unto this website. Can't you just put one of those picture-code-a-ma-bobs when creating a new nation?

The scripts were using the APIs, which we created to support third-party sites like NSEconomy. We're fine with that, if they pace themselves. We just need them to avoid smashing us with requests.

The site is still a little slow; the server's okay now but we're a little bottlenecked on bandwidth.

[Myakka]

well I think I'm seeing banners for the first time in the head but maybe they were always there.

[Collective Systems]

Perhaps you as an administrator can make a (more or less) official NationStates calculator.

[[violet]]

Perhaps you as an administrator can make a (more or less) official NationStates calculator.

NSEconomy is great; it just needs to spread its requests a bit better.

[Collective Systems]

Well, I mean, if you do it yourself you won't ever have problems with third parties. You can post national information on one's nation page, and make another (separate) calculator on the left tab bar.

[Charlotte Ryberg]

I can report that NSDossier has also been blocked.

Here is my thought on the situation: I was thinking about asking the web masters of the said sites to cache a copy of the database so that the site can look up its own copy rather than having to dig through the database here.

For example, a user looks up a region and the site makes a cache of the results on their database and that record expires at the next update (0500 and 1700 BST respectively) to keep it up to date. For example, say that at 0600 one user calls up a record of a region and NSEconomy searches, returns and stores the result in their cache until 1700, and when the next user does the same search at say 0601, NSEconomy returns the result from their cache.

[Commerce Heights]

I'm not sure exactly how NSE works, but possibly it waits until someone types in a region name, then hits us up for data on every nation in that region very quickly. And possibly it's not storing that data, either, so if one person looks up The Pacific three times in a minute, NSEconomy tries to load every Pacific nation three times in quick succession.

If that's the case, NSE needs to be smarter about how it gets data from us. It's fine to trawl the site slowly, but hitting us with a flood of requests all of a sudden ties up the server, slowing it for everyone else.

We were running pretty slow but seem to be back to normal now.

The script isn’t quite that badly designed —the problem appears to have been caused by a very large number of hits from a search engine bot. I’ve blocked bots from my site, and have also reduced the number of nationdata requests a session (page load) can make. If it continues to cause problems, I can take more extreme measures…

[Glen-Rhodes]

Here is my thought on the situation: I was thinking about asking the web masters of the said sites to cache a copy of the database so that the site can look up its own copy rather than having to dig through the database here.

I believe this is how NSTracker works. It requests data from the server once per day, then stores the info in an SQL database. Since NSTracker is open-source, it shouldn't be a problem for NSEconomy and NSDossier to switch to a similar system, if they don't already have one. Though, that completely depends on whether development of those tools is still active.

Edit: See post above about NSEconomy, I guess.

[[violet]]

I'm not sure exactly how NSE works, but possibly it waits until someone types in a region name, then hits us up for data on every nation in that region very quickly. And possibly it's not storing that data, either, so if one person looks up The Pacific three times in a minute, NSEconomy tries to load every Pacific nation three times in quick succession.

If that's the case, NSE needs to be smarter about how it gets data from us. It's fine to trawl the site slowly, but hitting us with a flood of requests all of a sudden ties up the server, slowing it for everyone else.

We were running pretty slow but seem to be back to normal now.

The script isn’t quite that badly designed —the problem appears to have been caused by a very large number of hits from a search engine bot. I’ve blocked bots from my site, and have also reduced the number of nationdata requests a session (page load) can make. If it continues to cause problems, I can take more extreme measures…

Thanks for making changes! I'll unblock the IP.

[[violet]]

Hmm, unblocked and got 23 requests from it in 10 seconds.

[Collective Systems]

"Nothing to see here. Move along."

[Muravyets]

im gonna go out on a limb here and claim its the Liberal NationStates thing...as the second unidentified

"Aprilway Oolsfay Ayday."

Oh, right, it's that time again. The year's least amusing holiday. *sigh* Thanks for clearing that up for me.

[[violet]]

Behavior doesn't seem to have changed much... I'll see no requests for a little while, then a burst of 10-25 requests within 5 or 10 seconds. Those bursts are going to hurt us during peak traffic times and really need to be spread out more.

[Commerce Heights]

Hmm, unblocked and got 23 requests from it in 10 seconds.

*checks logs* Apparently that search engine bot is continuing to crawl the site, having not rechecked the robots.txt since I changed it. Perhaps you should leave the site blocked for a while.

[Collective Systems]

"Nothing to see here. Move along."