Scripts banned

[[violet]]

Apparently that search engine bot is continuing to crawl the site, having not rechecked the robots.txt since I changed it. Perhaps you should leave the site blocked for a while.

OK. Any Game Mod can unblock it when you think you've got it under control.

Ideally you'd cap the number of requests per second that can come from any source, because if you just block bots, you might still flood us when several players use NSE at once.

[Collective Systems]

Can't you just reroute the site to a different (closed to members) page of NationStates that actively archives all national information?

[NewTexas]

Well, we don't know what happened to the XML feeds, but apparently NSEconomy is not the only one it affected.

NSDossier is dead too. It apparently happened around 3:17pm (GMT-7).

While the link http://www.nationstates.net/cgi-bin/nationdata.cgi/nation=newtexas works, it throws errors when we parse it.

Ah, we suspect it is this:

Code: Select all

<CATEGORY>
  <span class="afstrikeout">Civil Rights Lovefest</span>
  <span class="aflabel1">Nation-Hating Hippies</span>
</CATEGORY>

Was not counting on that in our schema!

Well, we are not fixing it for ~20 more hours...



BTW, the NSDossier user agent clearly idfentifies us if you have a prob:

Code: Select all

NSDossier Spider(NewTexas@*****.rr.com)

[[violet]]

Sorry about that, XML feed fixed.

[Khorata]

im gonna go out on a limb here and claim its the Liberal NationStates thing...as the second unidentified

"Aprilway Oolsfay Ayday."

Oh, right, it's that time again. The year's least amusing holiday. *sigh* Thanks for clearing that up for me.

Pig Latin. It's the time of year when the pigs who speak Pig Latin return to Springfield... But happy April Fools Eve.

[NewTexas]

Sorry about that, XML feed fixed.

Thanks [violet].

Regrettably that did not fix whatever is causing error:

Code: Select all

An error has occurred while opening external DTD 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd': The remote server returned an error: (503) Server Unavailable.

We encountered this once before and explicitly prohibited from looking up that DTD and it has been fine ever since.

Did anything else change about the feeds today by any chance?

Has the header:

Code: Select all

<?xml version="1.0" encoding="iso-8859-1"?>

had the encoding attribute before today?

Just trying to make sure nothing changed on this end before attacking our hosting service like we did when the

Code: Select all

<NATIONS>---MANY---</NATIONS>

turned into a list of 7000+ nations for Lazarus last month and blew our program.

[Fit battion]

Have you found what the other script is? Because I might have an inclination but I could be way off, need to talk to the owner of it first.

[Commerce Heights]

The bots have all gone bye-bye; I think it’s safe to unblock NSEconomy now.

[[violet]]

The bots have all gone bye-bye; I think it’s safe to unblock NSEconomy now.

Unblocked, but you still need to rate-limit your access to the API. When I entered "Testlandia," NSE sent us 22 GET requests in 7 seconds (querying Testlandia, its region The East Pacific, then 20 TEP nations in I think alphabetical order). I entered "Testlandia" again and it did the same thing again, re-looking up Testlandia and The East Pacific and 20 more nations, although this time the 20 nations were different, as it does seem to remember the first 20 at least for a while.

This will knock our server around if you have several people looking up nations at once during peak traffic times. The issue is not the number of requests but how you send them in bursts. I'd recommend you only look up the nation & region upon a user request, and otherwise use data gathered by a background trawl. I think some other scripters here do this and can probably help out with advice.

[[violet]]

Regrettably that did not fix whatever is causing error:

Code: Select all

An error has occurred while opening external DTD 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd': The remote server returned an error: (503) Server Unavailable.

I don't think that has anything to do with us. It looks like you're trying and failing to load a document from w3.org.

Did anything else change about the feeds today by any chance?

Nope, at least not intentionally. Same encoding, etc.

I don't know what that second script was (it sends a blank user agent) but if you're blocked you'll get 403 Forbidden errors from us.

[[violet]]

The issue is not the number of requests but how you send them in bursts. I'd recommend you only look up the nation & region upon a user request, and otherwise use data gathered by a background trawl. I think some other scripters here do this and can probably help out with advice.

Also NSE doesn't seem to remember what it's already scraped very well. I just saw a burst of requests that looked up the same 9 nations twice each in 6 seconds.

[Commerce Heights]

Unblocked, but you still need to rate-limit your access to the API. When I entered "Testlandia," NSE sent us 22 GET requests in 7 seconds (querying Testlandia, its region The East Pacific, then 20 TEP nations in I think alphabetical order). I entered "Testlandia" again and it did the same thing again, re-looking up Testlandia and The East Pacific and 20 more nations, although this time the 20 nations were different, as it does seem to remember the first 20 at least for a while.

This will knock our server around if you have several people looking up nations at once during peak traffic times. The issue is not the number of requests but how you send them in bursts. I'd recommend you only look up the nation & region upon a user request, and otherwise use data gathered by a background trawl. I think some other scripters here do this and can probably help out with advice.

Somehow the code that was supposed to rate-limit requests (in each session, not overall) was broken, possibly by a new version of PHP. I believe it’s working as intended now.

[[violet]]

Somehow the code that was supposed to rate-limit requests (in each session, not overall) was broken, possibly by a new version of PHP. I believe it’s working as intended now.

Thanks! Definitely seeing slower requests now.

You might want to think about building those region stats (e.g. Smallest GDP) in the background, rather than on request, since I think at the moment it's (a) very slow to calculate, and (b) inaccurate, if the 20 nations loaded is only a fraction of the region total.

[NewTexas]

So, is anybody being blocked now? In particular, 66.186.8.221 or maybe 209.132.230.21

Our Hosting Sericve has been looking into why NSDossier has not been working for 5 days now and the best they can come up with is that our IP address is being blocked. This is a halfway decent theory since putting the source code on 3 different servers shows no problem.

[NewTexas]

Is there something wrong with our script or did we just get caugth is a crossfire of mass bannings?

Thank you for looking into it.

[[violet]]

So, is anybody being blocked now? In particular, 66.186.8.221 or maybe 209.132.230.21

Neither of those two are blocked, but the one block still outstanding is very similar, so it's probably you. It sends a blank user agent so I can't be sure.

It's blocked because it periodically hits us with a flood of requests. E.g.:

Code: Select all

209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=winnipeg HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=obstinacy HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=natopia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=eta_carinae HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=righteous_lefties HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=senexia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=italia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=tir HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=nag_ehgoeg HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=kako HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=terraus HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=weirdville HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=egeden HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=the_zocalo HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=ineptia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=libby_the_lab HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=quemmelande HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=armacor HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=pendulous HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=turconia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=rlyeh HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=zimbada HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=argesea HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=eureka HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=look_at_me HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=alpha_centauri HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=teddymania HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=sherlockazulu HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=abu_ammar HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=techno HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=south_yorkshire HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=paradise HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=uberwald HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=koffee HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=kharkov HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=midlands HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=dalioranium HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=craven HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=gras HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=ilek-vaad HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=borkistan HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=crystal_palais HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=demoness HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=eli HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=cthonian_wasteland HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=nianacio HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=florida HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=ochakov HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=serandum HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=malawi HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=frappuccino HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=moneystan HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=sketch HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=feynland HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=architeuthis HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=kyupaa HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=korinekia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=west_siders HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=praetor HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=second_helpings HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=the_nine HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=xen HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=mathematicians HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=linne_weg HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=nak_nak HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=the_bruce HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=gracieland HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=ltlaliens HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=cuatela HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=vermin HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=blackbird HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=ballotonia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=lenninspart HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=myrth HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=moltar HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=equus HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=al_amarja HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=star_city HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=trashcan HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=haldana HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=sehvekah HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=emory HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=stevenageboro HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=latrinia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=lorthellian HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=gross_deutschland HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=augustalia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=purdunamis HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=orpheus HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=oxymoronia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=sunset HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=transtopia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=aston HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=sileetris HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=harponia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=rhinara HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=mariusgrad HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=japanada HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=lamoni HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=devlinia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=newtexas HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=go_canucks_go HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=united_faiths HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=the_night_elves HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=cogitation HTTP/1.1" 302 - "-" "-"


That's over 100 requests in one second. In this burst it sent us 500 requests over 6 seconds. That's well above our tolerance.

There are many bursts like this, where we get no requests from that IP for hours, then hundreds within a few seconds.

[Coffee and Crack]

On this note, I'm curious as to what are acceptable scripting policies against the server load, also is there a preferred time to do data mining? I wouldn't mind taking some time to rewrite the script to have a more acceptable caching policy.

[[violet]]

On this note, I'm curious as to what are acceptable scripting policies against the server load, also is there a preferred time to do data mining? I wouldn't mind taking some time to rewrite the script to have a more acceptable caching policy.

I guess ideally you'd scrape outside peak time (American late afternoon/evening), but it doesn't really matter so long as you space out requests. We can easily handle 1 request per second from you all day long (86,400 per day). What cause trouble are bursts of requests, 10 or 20 or 50 within a few seconds.

Wrong way to do it: Wait until a user wants information on a region, then attempt to load the region and every nation in it, even if that means 4,000 residents of Lazarus. Calculate, display page, throw away all the data you just scraped from NS.

Right way to do it: If you need stats from every nation in the game to display a page, scrape that slowly in the background and store them in your own DB, where it can be re-used. When someone wants to look up a nation or region, only scrape fresh information from NS that you actually need to be up-to-the-second, e.g. the actual region and/or nation being looked up (not all its neighbors).

I'm happy to tweak the API to make it more usable for third-party sites if that helps you get the info you need in fewer requests.

[Winnipeg]

So, is anybody being blocked now? In particular, 66.186.8.221 or maybe 209.132.230.21

Neither of those two are blocked, but the one block still outstanding is very similar, so it's probably you. It sends a blank user agent so I can't be sure.

It's blocked because it periodically hits us with a flood of requests. E.g.:

Code: Select all

209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=winnipeg HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=obstinacy HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=natopia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=eta_carinae HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=righteous_lefties HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=senexia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=italia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=tir HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=nag_ehgoeg HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=kako HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=terraus HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=weirdville HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=egeden HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=the_zocalo HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=ineptia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=libby_the_lab HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=quemmelande HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=armacor HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=pendulous HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=turconia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=rlyeh HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=zimbada HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=argesea HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=eureka HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=look_at_me HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=alpha_centauri HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=teddymania HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=sherlockazulu HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=abu_ammar HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=techno HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=south_yorkshire HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=paradise HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=uberwald HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=koffee HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=kharkov HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=midlands HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=dalioranium HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=craven HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=gras HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=ilek-vaad HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=borkistan HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=crystal_palais HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=demoness HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=eli HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=cthonian_wasteland HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=nianacio HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=florida HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=ochakov HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=serandum HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=malawi HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=frappuccino HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=moneystan HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=sketch HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=feynland HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=architeuthis HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=kyupaa HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=korinekia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=west_siders HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=praetor HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=second_helpings HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=the_nine HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=xen HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=mathematicians HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=linne_weg HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=nak_nak HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=the_bruce HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=gracieland HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=ltlaliens HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=cuatela HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=vermin HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=blackbird HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=ballotonia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=lenninspart HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=myrth HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=moltar HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=equus HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=al_amarja HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=star_city HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=trashcan HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=haldana HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=sehvekah HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=emory HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=stevenageboro HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=latrinia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=lorthellian HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=gross_deutschland HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=augustalia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=purdunamis HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=orpheus HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=oxymoronia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=sunset HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=transtopia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=aston HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=sileetris HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=harponia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=rhinara HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=mariusgrad HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=japanada HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=lamoni HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=devlinia HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=newtexas HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=go_canucks_go HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=united_faiths HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=the_night_elves HTTP/1.1" 302 - "-" "-"
209.132.230.240 - - [05/Apr/2010:19:04:55 -0700] "GET /cgi-bin/nationdata.cgi/nation=cogitation HTTP/1.1" 302 - "-" "-"


That's over 100 requests in one second. In this burst it sent us 500 requests over 6 seconds. That's well above our tolerance.

There are many bursts like this, where we get no requests from that IP for hours, then hundreds within a few seconds.

You know, that might be one of my old Census apps from ages ago...but I don't have access to the particular host it was on anymore...

I'll take a looksee at some of my php scripts and see if it might be any of mine (I just get the idea it might be one of mine since my nation is at the very top.)

Although, now that I look at it, I don't think it was mine since it looks like it was hitting nations outside either of the regions I've ever called home.

And edit again: Nope, not me or any of my previous scripts...never heard of the webhost that IP belongs to.

[NewTexas]

Looking at the population of The World,

56,455 nations @ 1 per second = 940 minutes or 15.68 hours

That is almost not practical since it spans a rollover.

The thing that spurs "needing" to download all the nations in a region is the arbitrary GDP calculation created years ago that factors in every nation in a region's statistics.

We suspect this has not been a problem in 4 years since we created our tool is that all the nations for the regions with more than 300 nations were never listed. Now that they are available, whenever a nation from a Feeder wants to see their GDP, then all the nations are polled. Add to that multithreading and we see the problem. With 300 nations or less, it was never more than a minute and may be why you never noticed.

We are sorry.

How about if we fix our code so it no longer requests nations for any region with over 300 nations, then can we come back?

[[violet]]

Looking at the population of The World,

56,455 nations @ 1 per second = 940 minutes or 15.68 hours

That is almost not practical since it spans a rollover.

The thing that spurs "needing" to download all the nations in a region is the arbitrary GDP calculation created years ago that factors in every nation in a region's statistics.

We suspect this has not been a problem in 4 years since we created our tool is that all the nations for the regions with more than 300 nations were never listed. Now that they are available, whenever a nation from a Feeder wants to see their GDP, then all the nations are polled. Add to that multithreading and we see the problem. With 300 nations or less, it was never more than a minute and may be why you never noticed.

We are sorry.

How about if we fix our code so it no longer requests nations for any region with over 300 nations, then can we come back?

What I'm getting at is that you might productively look at separating the information you need into two categories: (1) data that must be up-to-the-second, and thus must be sourced from the API at the moment the user clicks a button on your site, and (2) data that does not need to be up-to-the-second (although it should be reasonably current).

I'm not sure exactly what calculations you do, but if you produce a Regional GDP, for example, that number is unlikely to change much from minute to minute, or even day to day. This is particularly true for large regions: if the Regional GDP of Lazarus was $10 trillion yesterday, I can safely assume it's about $10 trillion today, even if many nations have come and gone and answered issues in the meantime. I could load up all 4,250 residents and check, and load them again and again whenever anyone wants data on a Lazarus nation, but that's a great amount of work just to confirm that, inevitably, the result is very close to $10 trillion.

So for data that isn't time-critical, there are a couple of ways we can access and store it. One would be that each daily update, we total up the relevant stats for all resident nations, save it in the region file, and make it available in the regiondata API (e.g. <REGIONGDP>10000000000</REGIONGDP>.) That, I think, would give you the raw data you needed to produce your output without necessitating the loading of every nation in the region. The REGIONGDP figure would not be up-to-the-second; it would be based on the last region update. But most of the time it would be very accurate--in large regions, it would be extremely accurate, since their mass makes them less likely to fluctuate.

That would be a solution that allows you to continue work the same as always--i.e. sending API requests at the instant the player requests them, rather than building a local DB of retrieved data yourself. The downside would be that you are dependent on us coding in the exact figures you need.

A more flexible approach from your point of view, should you have the time and inclination, is to use a background process to load the nations and regions in question over time, and store the relevant data in your own DB, which you access upon a player's request. You might decide to do this for all regions with more than 10 resident nations, for example (there are only about 700 of those). So when a player comes along and types in "testlandia", a nation that resides in Lazarus, your site would look up Testlandia via nationdata, because you want up-to-the-second data, but use the already-stored data on Lazarus you've scraped over the last 48 hours to calculate Testlandia's share of Regional GDP, etc.

This also lets your site be more responsive, because it's only waiting on a single reply from our server before it can build its page.

We need to do something, because it's not okay to send us 100 requests per second, even if it's only for 3 seconds to load a 300-nation region. By way of comparison, under normal load we average of 45 requests per second (including the game and the forum), many of which are non-dynamic, e.g. images, which are quicker to process. Sending us 100 requests for dynamic pages in a second means a very significant increase in server load. If it comes at a time when the site is already busy, which yours do since they are triggered by users, it can temporarily exceed our capacity.

Let me know if any of this is unclear or unhelpful! I want to support third-party NS sites.

[[violet]]

I should mention that I haven't monitored script loads before, so it's not that things have suddenly become a problem but rather that I've started paying attention.

[Winnipeg]

Well, I know I'd be in favour of GDP information being handled by the server itself, since most of the calculators out there are based on NSEconomy, which in turn was originally based on some of mine and Jasque's Formulas and Code that we released on the Meritocracy site several years ago. Commerce Heights has of course imrpoved on those considerably.

If I recall correctly the NSEconomy code is open source, so I'm not sure how that would play into it's usability for the formulas but I'm sure something could be worked out that we could get something worked into the game's system. If it was calculated during the game's update (or even every second update along with the Population increases, since this is usually the bigger GDP changer on a day to day basis) and then available through the game's APIs (nationdata.cgi and regiondata.cgi) it should make the collection far far easier...

Might even be something you could enable in the regiondata.cgi file like the message archives (so it's not on by default), but if requested, the GDP of the region would be listed, and then there would be a section that showed each nation's GDP;GDPperCapita, in the same order as the nation names in the <nations>. That way all that data could be pulled in 1 single request, although it might be more load than you want the regiondata.cgi file to go through so I'm not sure.

Sorry for rambling on, just throwing some ideas out there.

Robin

[NewTexas]

The issue of storing the data in a database sounds like the way to go. We could split the application into two distinct parts - one to serve up the data and one to collect the data, such as a routine that downloaded a nation a second until they had all been done once a day and each pull would either add, update or delete a database record. The problem there, is the need for a "catalog" or master XML list of all the nations (and one for all the regions). Otherwise, the only way to get that information is to screen scrape The World pages in the live site. We suggested this back on this post.

We could then limit the hits on the feeds to two - one to look up the nation requested (to get their region) and one to look up the region (to get the nations in the region). All the rest of the nation data could be collected from the database and if any are not found (i.e.: a new one), then that one could be gotten from the feeds as well. Unless someone created many nations and immediately looked them up, the traffic would be imperceptible. We can do this.

The possibility of adding the four or five "custom" or calculated fields to the feeds would be more than acceptable! We never coded a quadratic equation before and hope never to again. The GDP calculation on nations and the GRP calculations on regions would be awesome, but, we caution you that it is not an task to be taken lightly (even with the free PHP source code at hand). Of course, if the other components that went into that calculation were made available as well, that would be a bonus (things like: unemployment, GDP per capita, GRP per capita, exports, imports etc). But that is Ok if you want to skip it, both NSEconomy & NSDossier already have the calculations in place.

This is a new paradigm we have never considered. We used to do everything by throwing the feed data into memory and crunching it. We can certainly use a database though.

If that is acceptable, we will do a major rewrite. This method would be a way to still be able to serve up the GRP stats for the Feeders or even Lazarus. The only thing we need to know is if you can or want to produce the two catalog files for nations and regions or whether we need to scrape it off the pages of The World. It would really be nice if you dumped the same fields on The World listings for nations (date, nation, WA status, category and motto) and for regions (date, region, number of nations and delegate). But, just names of nations and names of regions would be ok. We are thinking we would shoot for the stars if we are asking for something. We have an ulterior purpose as well, we have use those fields to populate our NSHistory database and would save us from having to crawl those pages scraping data off them.

In the meantime, we have shutdown NSDossier. We mention this because we would like our IP addresses be unblocked. Our map site shares the same host and none of our maps work without the feeds (yes, poor design, but it is a legacy app that we don't feel like rewriting).

Thank you.


edit: We can send you the URL the closed beta of NSHistory if you are interested. Is that admin email address right? If not, you can TG us on NS.

[[violet]]

Unbanned 209.132.230.240.

I'll get back to you about the rest; just going over our API stats atm.