NATION

PASSWORD

Scripts banned

Bug reports, general help, ideas for improvements, and questions about how things are meant to work.
User avatar
[violet]
Site Admin
 
Posts: 15723
Founded: Antiquity

Scripts banned

Postby [violet] » Wed Mar 31, 2010 3:36 pm

I just had to ban two script IPs that were hitting nationdata.cgi at an unacceptably high rate (multiple times per second). One identified itself as NSEconomy and the other did not identify itself. Please let me know if you own these scripts and are now getting "403 Forbidden" errors.

User avatar
East Fancainia
Negotiator
 
Posts: 6068
Founded: Dec 05, 2009
Ex-Nation

Postby East Fancainia » Wed Mar 31, 2010 3:40 pm

[violet] wrote:I just had to ban two script IPs that were hitting nationdata.cgi at an unacceptably high rate (multiple times per second). One identified itself as NSEconomy and the other did not identify itself. Please let me know if you own these scripts and are now getting "403 Forbidden" errors.

Why would NSE be doing that? Could this be the cause of my computer running slow on Nationstates for like the past minute?
Last edited by East Fancainia on Wed Mar 31, 2010 3:42 pm, edited 1 time in total.

User avatar
Collective Systems
Ambassador
 
Posts: 1503
Founded: Feb 25, 2010
Ex-Nation

Postby Collective Systems » Wed Mar 31, 2010 3:41 pm

I was getting that error a few minutes ago, but not anymore. This website still seems a tad slow, however.

User avatar
Unidox
Minister
 
Posts: 2592
Founded: Jan 25, 2004
New York Times Democracy

Postby Unidox » Wed Mar 31, 2010 3:46 pm

East Fancainia wrote:
[violet] wrote:I just had to ban two script IPs that were hitting nationdata.cgi at an unacceptably high rate (multiple times per second). One identified itself as NSEconomy and the other did not identify itself. Please let me know if you own these scripts and are now getting "403 Forbidden" errors.

... Could this be the cause of my computer running slow on Nationstates for like the past minute?


I've noticed this too. Although, forum lag time usually has something to do with the server, or the number of Ad Bots on today.
Caninope wrote:It's NSG. The 20th Circle of LIMBO!

Buffett and Colbert wrote:Always here to ruin the day. 8)

Living Freedom Land wrote:Oh, so now you want gay people to take part in the sacred institution of tax rebates too? You liberals sicken me.

Lacadaemon wrote:I mean, hell, in a properly regulated market, pension stripping schemes like Zynga wouldn't ever have a sniff of an IPO (see Groupon). But it's all wild westy now. Lie down with dogs and so forth.

User avatar
Frenca
Minister
 
Posts: 2991
Founded: Feb 14, 2010
Ex-Nation

Postby Frenca » Wed Mar 31, 2010 3:46 pm

[violet] wrote:I just had to ban two script IPs that were hitting nationdata.cgi at an unacceptably high rate (multiple times per second). One identified itself as NSEconomy and the other did not identify itself. Please let me know if you own these scripts and are now getting "403 Forbidden" errors.

That's probably the reason why my computer was acting weird here...
The Great Kingdom of Frenca and the Protectorates of The Kingdom of Auatris, The Kingdom of France-Bourbon, and The Kingdom of France and Bourbon
Factbook 1: viewtopic.php?f=23&t=40250 New Factbook!: viewtopic.php?f=23&t=69302
Old Embassies: viewtopic.php?f=23&t=37051 Embassies: viewtopic.php?f=23&t=44956
My villa. Resting in the archives.
Economic Left/Right: 0.88
Social Libertarian/Authoritarian: -3.44

User avatar
[violet]
Site Admin
 
Posts: 15723
Founded: Antiquity

Postby [violet] » Wed Mar 31, 2010 3:47 pm

I'm not sure exactly how NSE works, but possibly it waits until someone types in a region name, then hits us up for data on every nation in that region very quickly. And possibly it's not storing that data, either, so if one person looks up The Pacific three times in a minute, NSEconomy tries to load every Pacific nation three times in quick succession.

If that's the case, NSE needs to be smarter about how it gets data from us. It's fine to trawl the site slowly, but hitting us with a flood of requests all of a sudden ties up the server, slowing it for everyone else.

We were running pretty slow but seem to be back to normal now.

User avatar
Schultaria Prime
Envoy
 
Posts: 250
Founded: Mar 01, 2004
Left-wing Utopia

Postby Schultaria Prime » Wed Mar 31, 2010 3:48 pm

Don't know why, but when I logged on to NSEconomy I received economic data for over 15,000 nations (including one that I let die a couple of weeks ago) instead of the usual welcome screen. As far as I can recollect NSEconomy should have built in brakes from accessing economic data from feeder regions and other large entities, but it seems like something is quite wrong with the website at the moment.

(CORRECTION) It's when I access NSEconomy through a search engine that it seems to do this. Otherwise, it loads up normally.
Last edited by Schultaria Prime on Wed Mar 31, 2010 3:54 pm, edited 3 times in total.
The United Socialist States of Schultaria Prime (USSSP)

Information: Military White Paper
Storefront: Designwerks Heavy Industry | Designwerks Interkosmos | Designwerks Computation
Active Storylines:


User avatar
Collective Systems
Ambassador
 
Posts: 1503
Founded: Feb 25, 2010
Ex-Nation

Postby Collective Systems » Wed Mar 31, 2010 3:52 pm

Na, it's still a bit slow for me.

I don't understand why you allow bots unto this website. Can't you just put one of those picture-code-a-ma-bobs when creating a new nation?

User avatar
Redneck United States
Civil Servant
 
Posts: 7
Founded: Sep 17, 2009
Ex-Nation

Postby Redneck United States » Wed Mar 31, 2010 4:02 pm

im gonna go out on a limb here and claim its the Liberal NationStates thing...as the second unidentified

User avatar
Collective Systems
Ambassador
 
Posts: 1503
Founded: Feb 25, 2010
Ex-Nation

Postby Collective Systems » Wed Mar 31, 2010 4:05 pm

Redneck United States wrote:im gonna go out on a limb here and claim its the Liberal NationStates thing...as the second unidentified

"Aprilway Oolsfay Ayday."

User avatar
[violet]
Site Admin
 
Posts: 15723
Founded: Antiquity

Postby [violet] » Wed Mar 31, 2010 4:12 pm

Collective Systems wrote:Na, it's still a bit slow for me.

I don't understand why you allow bots unto this website. Can't you just put one of those picture-code-a-ma-bobs when creating a new nation?

The scripts were using the APIs, which we created to support third-party sites like NSEconomy. We're fine with that, if they pace themselves. We just need them to avoid smashing us with requests.

The site is still a little slow; the server's okay now but we're a little bottlenecked on bandwidth.

User avatar
Myakka
Diplomat
 
Posts: 675
Founded: Jul 11, 2007
Ex-Nation

Postby Myakka » Wed Mar 31, 2010 4:17 pm

well I think I'm seeing banners for the first time in the head but maybe they were always there.

User avatar
Collective Systems
Ambassador
 
Posts: 1503
Founded: Feb 25, 2010
Ex-Nation

Postby Collective Systems » Wed Mar 31, 2010 4:22 pm

Perhaps you as an administrator can make a (more or less) official NationStates calculator.

User avatar
[violet]
Site Admin
 
Posts: 15723
Founded: Antiquity

Postby [violet] » Wed Mar 31, 2010 4:26 pm

Collective Systems wrote:Perhaps you as an administrator can make a (more or less) official NationStates calculator.

NSEconomy is great; it just needs to spread its requests a bit better.

User avatar
Collective Systems
Ambassador
 
Posts: 1503
Founded: Feb 25, 2010
Ex-Nation

Postby Collective Systems » Wed Mar 31, 2010 4:28 pm

Well, I mean, if you do it yourself you won't ever have problems with third parties. You can post national information on one's nation page, and make another (separate) calculator on the left tab bar.

User avatar
Charlotte Ryberg
The Muse of the Westcountry
 
Posts: 15007
Founded: Mar 14, 2007
Civil Rights Lovefest

Postby Charlotte Ryberg » Wed Mar 31, 2010 4:40 pm

I can report that NSDossier has also been blocked.

Here is my thought on the situation: I was thinking about asking the web masters of the said sites to cache a copy of the database so that the site can look up its own copy rather than having to dig through the database here.

For example, a user looks up a region and the site makes a cache of the results on their database and that record expires at the next update (0500 and 1700 BST respectively) to keep it up to date. For example, say that at 0600 one user calls up a record of a region and NSEconomy searches, returns and stores the result in their cache until 1700, and when the next user does the same search at say 0601, NSEconomy returns the result from their cache.

User avatar
Commerce Heights
Minister
 
Posts: 2050
Founded: Antiquity
Anarchy

Postby Commerce Heights » Wed Mar 31, 2010 4:42 pm

[violet] wrote:I'm not sure exactly how NSE works, but possibly it waits until someone types in a region name, then hits us up for data on every nation in that region very quickly. And possibly it's not storing that data, either, so if one person looks up The Pacific three times in a minute, NSEconomy tries to load every Pacific nation three times in quick succession.

If that's the case, NSE needs to be smarter about how it gets data from us. It's fine to trawl the site slowly, but hitting us with a flood of requests all of a sudden ties up the server, slowing it for everyone else.

We were running pretty slow but seem to be back to normal now.

The script isn’t quite that badly designed :P—the problem appears to have been caused by a very large number of hits from a search engine bot. I’ve blocked bots from my site, and have also reduced the number of nationdata requests a session (page load) can make. If it continues to cause problems, I can take more extreme measures…

User avatar
Glen-Rhodes
Powerbroker
 
Posts: 9027
Founded: Jun 25, 2008
Civil Rights Lovefest

Postby Glen-Rhodes » Wed Mar 31, 2010 4:44 pm

Charlotte Ryberg wrote:Here is my thought on the situation: I was thinking about asking the web masters of the said sites to cache a copy of the database so that the site can look up its own copy rather than having to dig through the database here.

I believe this is how NSTracker works. It requests data from the server once per day, then stores the info in an SQL database. Since NSTracker is open-source, it shouldn't be a problem for NSEconomy and NSDossier to switch to a similar system, if they don't already have one. Though, that completely depends on whether development of those tools is still active.

Edit: See post above about NSEconomy, I guess.
Last edited by Glen-Rhodes on Wed Mar 31, 2010 4:45 pm, edited 1 time in total.

User avatar
[violet]
Site Admin
 
Posts: 15723
Founded: Antiquity

Postby [violet] » Wed Mar 31, 2010 4:46 pm

Commerce Heights wrote:
[violet] wrote:I'm not sure exactly how NSE works, but possibly it waits until someone types in a region name, then hits us up for data on every nation in that region very quickly. And possibly it's not storing that data, either, so if one person looks up The Pacific three times in a minute, NSEconomy tries to load every Pacific nation three times in quick succession.

If that's the case, NSE needs to be smarter about how it gets data from us. It's fine to trawl the site slowly, but hitting us with a flood of requests all of a sudden ties up the server, slowing it for everyone else.

We were running pretty slow but seem to be back to normal now.

The script isn’t quite that badly designed :P—the problem appears to have been caused by a very large number of hits from a search engine bot. I’ve blocked bots from my site, and have also reduced the number of nationdata requests a session (page load) can make. If it continues to cause problems, I can take more extreme measures…

Thanks for making changes! I'll unblock the IP.

User avatar
[violet]
Site Admin
 
Posts: 15723
Founded: Antiquity

Postby [violet] » Wed Mar 31, 2010 4:48 pm

Hmm, unblocked and got 23 requests from it in 10 seconds.

User avatar
Collective Systems
Ambassador
 
Posts: 1503
Founded: Feb 25, 2010
Ex-Nation

Postby Collective Systems » Wed Mar 31, 2010 4:49 pm

"Nothing to see here. Move along."
Last edited by Collective Systems on Wed Mar 31, 2010 5:10 pm, edited 1 time in total.

User avatar
Muravyets
Postmaster-General
 
Posts: 12755
Founded: Aug 18, 2005
Ex-Nation

Postby Muravyets » Wed Mar 31, 2010 4:51 pm

Collective Systems wrote:
Redneck United States wrote:im gonna go out on a limb here and claim its the Liberal NationStates thing...as the second unidentified

"Aprilway Oolsfay Ayday."

Oh, right, it's that time again. The year's least amusing holiday. *sigh* Thanks for clearing that up for me.
Kick back at Cafe Muravyets
And check out my other RP, too. (Don't take others' word for it -- see for yourself. ;) )
I agree with Muravyets because she scares me. -- Verdigroth
However, I am still not the topic of this thread.

User avatar
[violet]
Site Admin
 
Posts: 15723
Founded: Antiquity

Postby [violet] » Wed Mar 31, 2010 4:54 pm

Behavior doesn't seem to have changed much... I'll see no requests for a little while, then a burst of 10-25 requests within 5 or 10 seconds. Those bursts are going to hurt us during peak traffic times and really need to be spread out more.

User avatar
Commerce Heights
Minister
 
Posts: 2050
Founded: Antiquity
Anarchy

Postby Commerce Heights » Wed Mar 31, 2010 4:54 pm

[violet] wrote:Hmm, unblocked and got 23 requests from it in 10 seconds.

*checks logs* Apparently that search engine bot is continuing to crawl the site, having not rechecked the robots.txt since I changed it. Perhaps you should leave the site blocked for a while. :?

User avatar
Collective Systems
Ambassador
 
Posts: 1503
Founded: Feb 25, 2010
Ex-Nation

Postby Collective Systems » Wed Mar 31, 2010 4:56 pm

"Nothing to see here. Move along."
Last edited by Collective Systems on Wed Mar 31, 2010 5:10 pm, edited 1 time in total.

Next

Advertisement

Remove ads

Return to Technical

Who is online

Users browsing this forum: No registered users

Advertisement

Remove ads