Questions about SSE changes and API limits

[New Rogernomics]

I am gradually working on setting it up for SSE and waiting for the next WA resolution to drop so I can switch all the resolution stuff to SSE, as I don't have an example of what it shows in the SSE data for that yet, except for the votes and such.

1. Realistically if "API Rate Limit: 50 requests per 30 seconds" is considered too high comparably with SSE, should we really follow "API Rate Limit: 50 requests per 30 seconds" for bot/script use, or use a lot higher limit if there is a request?

2. What technically should be phased out entirely or considered illegal for bot/script use regarding the shards and api.cgi requests in general?

Scenario 1: A discord bot/script uses https://www.nationstates.net/cgi-bin/ap ... testlandia or https://www.nationstates.net/cgi-bin/ap ... ted_realms. This I figure can be covered through the daily dumps + SSE as an updater. [But] If bot/script is offline for server maintenance, then a look up for nation or region would fail in the interval of time till the next daily dump. So potentially this is where the API still comes in use even with SSE in existence.

Scenario 2: A discord bot/script uses the cards api to look up a card for a user when they click a button. Potentially can put all the card seasons into a database like I do with daily dumps, then reference that, but it would require bot/script users to carry a lot more GBs on disk, especially as more card seasons come out over time.

Scenario 3: A discord bot/script uses the https://www.nationstates.net/cgi-bin/ap ... =proposals, in order to collect the data for proposals that have dropped, and how quorum is going.

Scenario 4: A discord bot/script uses the https://www.nationstates.net/cgi-bin/ap ... resolution, similar to 3, but with the exception of finding who has voted for/against the resolution as opposed to quorum.

Scenario 5: A discord bot/script allows a user to verify their nation, set an x auto login, and then push a button to do something like add/edit a dispatch with new content, look up/answer an issue, or adopt a resolution.

I already had a daily dumps + SSE set up in my own bot, and so it is just a painstaking process of going through and switching off anything that might have used the shards and such as a backup or data verify step.

These are the remaining bits of confusion I have even having read the API docs over, but otherwise really looking forward to what happens with the SSE.

[[violet]]

Thanks for looking at SSE!

The Live Happenings SSE API isn't meant to be a general replacement for everything else. It's just for bots that want to monitor NS Happenings and react to them.

Some of what you describe above doesn't have anything to do with live monitoring, so you wouldn't benefit from migrating. If your bot doesn't want to be notified of live events as they happen, but instead wants to respond to a user request whenever that occurs, then go fetch relevant data, that still belongs on the older APIs. It may be very inefficient if it receives a LOT of user requests and each time it needs to go scrape the exact same data because it has no memory of what it did 10 milliseconds ago, but that's not a live-monitoring problem; that's a it's-time-for-a-cache problem.

Essentially:

• If your bot rapidly requests the same Happenings from NS over and over just to see if anything changed, it should use the Live Happenings SSE API instead. An example would be a discord bot that wants to notify people in a channel that something just happened on NS.
• If your bot doesn't do anything until someone asks it to look something up, it should probably use the regular APIs. But if it's very active, it should consider a cache.

Also: nobody cares about a few hundred requests per day. I do care when I see bots making making thousands / tens of thousands / hundreds of thousands requests per day, especially when it's for the same info, and/or seems obviously broken, e.g. every request is duplicated.

"API Rate Limit: 50 requests per 30 seconds" (which doesn't apply to the Live Happenings SSE API, since there you make one request and hold open the connection to receive a steady stream of events) is a crude metric that can't tell the difference between light and heavy requests. It is possible to stay within the rate limits and still make our servers cry, and possible to exceed them without causing the servers to skip a beat. So I can't really give good advice there.

[New Rogernomics]

Thanks for the quick response, Violet.

Good to know that the API will still exist for problems that can show up in-between the SSE and daily dumps, which is usually nation look up related, with the database import not keeping pace with the general requests in the bot/script and so going after the API when it doesn't actually need to - or when it would appear just a few seconds later.

Caching certainly helps and I've been doing that where possible, and I try to set a buffer in the global rate limiter, just in case.

Potentially though a server restart after a period offline can make a bot/script react like a cat or dog deprived of their supper and rush the API. But events where NationStates itself has been down have been useful for going after those bugs ironically.