Scripts banned

[[violet]]

Okay, I've written up some rate-limiting code for the API. Will drop it in tomorrow I think. It will automatically ban sources for 15 minutes if they send a flood of requests (more than 100/minute). That way you can experiment without risking getting locked out for hours or days.

There's a WinHttpRequest script on lunarmani.com hitting us up to half a million times per day, so that one will need some work too.

[NewTexas]

Unbanned 209.132.230.240.

I'll get back to you about the rest; just going over our API stats atm.

Thank you.

[Winnipeg]

I've disabled all of my feed scraping scripts for the time being (except the new Message Board Feed as that one already limits itself to 5 seconds between requests), so I can make sure they aren't excessively hitting the server.

[[violet]]

FYI, here are some stats on nationdata.cgi requests over the last 20 hours:

Code: Select all

    Hits IP Address
  137742 67.210.106.242
   46426 208.113.238.141
    8404 208.77.96.72
     940 67.195.112.181


[Winnipeg]

That last one is a Yahoo Crawler.

I'm somewhat ashamed to admit that the second one was me, but you should see that one drop substantially tonight.

[[violet]]

the second one was me

I noticed odd behavior from that one. It seems to have two flavors: a speed-limited version that queries "nationdata/" (with a trailing slash) once every 10 seconds, and an out-of-control version that queries "nationdata?" (with a trailing question mark) in wild bursts of dozens of requests per second.

[[violet]]

I've written up some rate-limiting code for the API.

Created a new thread to announce this: viewtopic.php?f=15&t=45424

[Winnipeg]

The rate limited one was probably my regular census app...the other one was a UN Endorsment type scripty thingy, which I've shut down now because I'm not sure how I would do up to date endo counts for a region like TWP without going way too hard on the API so as it stands right now I won't be ressurecting that one (already got the usual "Don't say it's so" or "OH NOES!" from my users )

Census one I will work on again but will be trying to slow it down some more just to make sure it's not beating anyone up.

[[violet]]

The census one (trailing slash) seems totally fine, from my eyeballing of the logs -- one request every 10 seconds is well within the limit (6/minute compared to 100/minute).

[Ballotonia]

I only run my scripts once in several weeks, so it took me a while to run into this one too. My appologies!

Got a "420 Unused Error" and a "421" as well, so it took me a while to figure out what the problem was. Fortunately, after some time the server functioned again and my now slowed down (1 Get per second max) scripts are working again.

So, if you see my IP in the log as bumping into the limit of >1 Get per second, please know the problem has already been fixed on my end.

Ballotonia

[Fostonia2]

Ah, so that's why NSTracker stopped getting all the nations during its daily data pull . I've already updated my NSTracker code to pull 50 records at a time, every 5 minutes, until it updates all of its trackers, just figured I'd let you know that my server was one of those IP's that might have been problematic.

[Fostonia2]

I've added a local database 8-hour cache to NSTracker for normal stats viewing (the historical graphs will still use info from the daily [midnight central time] pulls). NSTracker may not be popular enough to cause too many problems, but it couldn't hurt.

[Zemnaya Svoboda]

Not to be a gravedigger but I have a tool which I call curld which I made to throttle myself.

It's a C daemon which accepts requests on a port and makes the requests itself and then delivers the results to the requestor. Unfortunately it's currently without any kind of authentication or control. Perhaps I should simply move it to a local socket before publishing it to you guys.