[Idea] Feature Request: Endorsement Page

[Sandaoguo]

Given that the Script Rules for HTML interaction have always had an ambiguity problem, I'd rather not be staring down the barrel of a potential DEAT or WA ban because 5 years from now admins forget this thread and read the wording like I am.

I have to point out, this is the exact problem with bots on the HTML site, and why we're trying to shift them onto the API!

Branching off of this in the HTML Script Rules thread, I'd like to make a feature request. This could be an API feature, but I also think there's no game balance issues in making it a user-facing feature. A feature like this wouldn't be useful for R/D due to speed issues. if there is a concern there, the page doesn't have to be "live" and could instead be updated per major update.

Cross-Endorsement Page
Current state: Players have no way of knowing who they aren't endorsing, without cross-referencing every single WA member of their region with their endorsement list. Players rely on scripts that use HTML-scraping or API calls to generate lists for them.
Current dev issues: Scripts are relying on a complex combination of daily dumps, rate-limited API calls, and page scraping of things like Happenings to craft accurate lists. The most user-friendly tools require users to enable cross-site cookies to work.
Proposed feature: A Endorsements page where players can easily endorse all World Assembly members in their region. The page would ideally provide buttons that use AJAX calls to endorse a nation, rather than just generating a list of links that players have to them open individually to the nation page and endorse.
Example: What I'm envisioning is essentially the tool I've created placed directly in NationStates itself (which obviously wouldn't require providing localids or enabling cross-site cookies) - https://tspforums.xyz/archer/interactive.html

This would be preferable over an API call that endorses a nation. Because the API is rate limited by source, API-based tools would have restricted usability and likely be poorly adopted compared to the current state options.

[Dawn Denac]

I have to point out, this is the exact problem with bots on the HTML site, and why we're trying to shift them onto the API!

Branching off of this in the HTML Script Rules thread, I'd like to make a feature request. This could be an API feature, but I also think there's no game balance issues in making it a user-facing feature. A feature like this wouldn't be useful for R/D due to speed issues. if there is a concern there, the page doesn't have to be "live" and could instead be updated per major update.

Cross-Endorsement Page
Current state: Players have no way of knowing who they aren't endorsing, without cross-referencing every single WA member of their region with their endorsement list. Players rely on scripts that use HTML-scraping or API calls to generate lists for them.
Current dev issues: Scripts are relying on a complex combination of daily dumps, rate-limited API calls, and page scraping of things like Happenings to craft accurate lists. The most user-friendly tools require users to enable cross-site cookies to work.
Proposed feature: A Endorsements page where players can easily endorse all World Assembly members in their region. The page would ideally provide buttons that use AJAX calls to endorse a nation, rather than just generating a list of links that players have to them open individually to the nation page and endorse.
Example: What I'm envisioning is essentially the tool I've created placed directly in NationStates itself (which obviously wouldn't require providing localids or enabling cross-site cookies) - https://tspforums.xyz/archer/interactive.html

This would be preferable over an API call that endorses a nation. Because the API is rate limited by source, API-based tools would have restricted usability and likely be poorly adopted compared to the current state options.

Alternatively, why not just manually cross endorse people without scripts? At most you'd have to open a few extra nation links, click endorse and done. A site-coded version of this just sounds like a lot of work for something that is easily doable without scripts or automating it.

[Aurum Raider]

Branching off of this in the HTML Script Rules thread, I'd like to make a feature request. This could be an API feature, but I also think there's no game balance issues in making it a user-facing feature. A feature like this wouldn't be useful for R/D due to speed issues. if there is a concern there, the page doesn't have to be "live" and could instead be updated per major update.

Let me stop you right here. Yes, it is useful for R/D and particularly for defenders for whom pre-move cross-endorsing is a very common practice. This also would make post-move cross endorsing far easier, and crossing up in literally any context in R/D

This would be a change practically only ever useful to defenders. It would only barely be faster for endo-tarters who have all the time in the world to begin with.

[Refuge Isle]

Branching off of this in the HTML Script Rules thread, I'd like to make a feature request. This could be an API feature, but I also think there's no game balance issues in making it a user-facing feature. A feature like this wouldn't be useful for R/D due to speed issues. if there is a concern there, the page doesn't have to be "live" and could instead be updated per major update.

Cross-Endorsement Page
Current state: Players have no way of knowing who they aren't endorsing, without cross-referencing every single WA member of their region with their endorsement list. Players rely on scripts that use HTML-scraping or API calls to generate lists for them.
Current dev issues: Scripts are relying on a complex combination of daily dumps, rate-limited API calls, and page scraping of things like Happenings to craft accurate lists. The most user-friendly tools require users to enable cross-site cookies to work.
Proposed feature: A Endorsements page where players can easily endorse all World Assembly members in their region. The page would ideally provide buttons that use AJAX calls to endorse a nation, rather than just generating a list of links that players have to them open individually to the nation page and endorse.
Example: What I'm envisioning is essentially the tool I've created placed directly in NationStates itself (which obviously wouldn't require providing localids or enabling cross-site cookies) - https://tspforums.xyz/archer/interactive.html

This would be preferable over an API call that endorses a nation. Because the API is rate limited by source, API-based tools would have restricted usability and likely be poorly adopted compared to the current state options.

Alternatively, why not just manually cross endorse people without scripts? At most you'd have to open a few extra nation links, click endorse and done. A site-coded version of this just sounds like a lot of work for something that is easily doable without scripts or automating it.

There are more than 30 regions with over a hundred WA members, and nine with over five hundred. Endotarting in those regions is more than a few clicks, and constantly re-checking nations that you have missed, as well as new nations that have arrived or WA'd up, is not feasible without scripts.

[Sandaoguo]

Alternatively, why not just manually cross endorse people without scripts? At most you'd have to open a few extra nation links, click endorse and done. A site-coded version of this just sounds like a lot of work for something that is easily doable without scripts or automating it.

As explained when describing the current state, there isn't a way for players to know who they are and aren't endorsing, without manually cross-referencing the endorsements list of every single WA member in their region. Nobody is or is ever going to do that, especially in regions with hundreds of WA members.

Generally though, "why not do this manually" is not a valuable comment or suggestion on feature requests for things that are being done using "HTML site bots", page scraping, or inefficient API calls. The goal of the request is to eliminate the necessity of the scripts. Suggesting to just do things manually isn't going to eliminate scripts, since the reason the scripts exist is to fill the gap caused by players not wanting to do X manually. Any feature needs to be equal to or better than what a third-party tool can provide, otherwise users will just continue to use the third-party tools.

Branching off of this in the HTML Script Rules thread, I'd like to make a feature request. This could be an API feature, but I also think there's no game balance issues in making it a user-facing feature. A feature like this wouldn't be useful for R/D due to speed issues. if there is a concern there, the page doesn't have to be "live" and could instead be updated per major update.

Let me stop you right here. Yes, it is useful for R/D and particularly for defenders for whom pre-move cross-endorsing is a very common practice. This also would make post-move cross endorsing far easier, and crossing up in literally any context in R/D

Defenders are already successfully and easily cross-endorsing pre-move, there's little to no room for improvement there. Anyways, as I clearly pointed out in the OP, any R/D balance consideration can be addressed by making the page useless in R/D contexts, like only updating once per day (as my tool does, for example, since it relies 100% on daily dumps). The current state of endorsement scripts aren't targeted for R/D, but instead permanent residents of the region in an effort to get the community to endorse each other as much as possible. You can reference TSP's and TNP's WA Programs for example.

[Dawn Denac]

Alternatively, why not just manually cross endorse people without scripts? At most you'd have to open a few extra nation links, click endorse and done. A site-coded version of this just sounds like a lot of work for something that is easily doable without scripts or automating it.

There are more than 30 regions with over a hundred WA members, and nine with over five hundred. Endotarting in those regions is more than a few clicks, and constantly re-checking nations that you have missed, as well as new nations that have arrived or WA'd up, is not feasible without scripts.

Echoing Aurum's sentiment above me, there's also the fact this would have quite a significant impact on R/D. Going off my point, again, this seems like a lot of work for something that can be done manually or with some basic script without site-coding it. The work here in this case (for the staff team) seems to outweigh the gain for anyone else except big GCR/UCRs with, as you pointed out, over a hundred WA members or defenders on military operations.

[Sandaoguo]

... there's also the fact this would have quite a significant impact on R/D.

Please make sure you're fully reading posts before making comments, as any issues with R/D balance has already been addressed in the OP. This feature requests would not be introducing anything that existing third-party tools don't already do. It's about taking those third-party tools and making what they do a native feature, so admins and devs don't have to worry about HTML site scripting rules.

[Ever-Wandering Souls]

Making it require only trivial work for GCR's/Large UCR's is also inherantly an R&D thing, as the primary point of crossing in those cases is to increase total regional influence and make coups/raids harder to execute, especially in the face of GCR decay. This was split from a thread largely about banning attempted enhancements to the manual telegram system, and I honestly hold a similar view - some things *should* take a lot of time and effort, as the cost of payback. Crossing 1000 people is one of those. If semi-automating it is hacky, risky, or potentially illegal, then don't do it.

I have less objection to an "everyone currently endorsed by xxx" API call, at least, if technically feasible.

For that matter, though, if this was an HTML page with buttons, I would hope it would be bot-restricted similar to the Issues and telegrams pages per the other thread.

[Dawn Denac]

... there's also the fact this would have quite a significant impact on R/D.

Please make sure you're fully reading posts before making comments, as any issues with R/D balance has already been addressed in the OP. This feature requests would not be introducing anything that existing third-party tools don't already do. It's about taking those third-party tools and making what they do a native feature, so admins and devs don't have to worry about HTML site scripting rules.

I'm well aware of how to read, thank you. It doesn't change anything Aurum or I pointed out already, but Souls addressed what I was going to say as I was typing this, so. Cool stuff.

[Sandaoguo]

Making it require only trivial work for GCR's/Large UCR's is also inherantly an R&D thing, as the primary point of crossing in those cases is to increase total regional influence and make coups/raids harder to execute, especially in the face of GCR decay. This was split from a thread largely about banning attempted enhancements to the manual telegram system, and I honestly hold a similar view - some things *should* take a lot of time and effort, as the cost of payback. Crossing 1000 people is one of those. If semi-automating it is hacky, risky, or potentially illegal, then don't do it.

Again, please realize that this is not introducing anything that's not already being done with third-party tools. These tools are not illegal, as [v] already clarified if you read or click-thru to the quoted post in the OP. I'm proposing to NS devs that they take something third-party devs are already doing using a combination of methods (page scraping, daily dumps, API calls), and in the interest of eliminating the third-party scripts altogether, making it a native feature.

There's really no point in discussing R/D balance or whatever as if this is a new idea request, since there are a plethora of third-party tools that do this already. The point, to repeat myself, is to make it a native feature and eliminate the scripts. If you want to discuss R/D balance, I would suggest opening a separate thread to propose a scripting rules change, as bogging this thread down with that discussion won't change that third-party tools are already doing all of this.

[Ever-Wandering Souls]

Making it require only trivial work for GCR's/Large UCR's is also inherantly an R&D thing, as the primary point of crossing in those cases is to increase total regional influence and make coups/raids harder to execute, especially in the face of GCR decay. This was split from a thread largely about banning attempted enhancements to the manual telegram system, and I honestly hold a similar view - some things *should* take a lot of time and effort, as the cost of payback. Crossing 1000 people is one of those. If semi-automating it is hacky, risky, or potentially illegal, then don't do it.

Again, please realize that this is not introducing anything that's not already being done with third-party tools. These tools are not illegal, as [v] already clarified if you read or click-thru to the quoted post in the OP. I'm proposing to NS devs that they take something third-party devs are already doing using a combination of methods (page scraping, daily dumps, API calls), and in the interest of eliminating the third-party scripts altogether, making it a native feature.

There's really no point in discussing R/D balance or whatever as if this is a new idea request, since there are a plethora of third-party tools that do this already. The point, to repeat myself, is to make it a native feature and eliminate the scripts. If you want to discuss R/D balance, I would suggest opening a separate thread to propose a scripting rules change, as bogging this thread down with that discussion won't change that third-party tools are already doing all of this.

I don't think "scripts do this so it should be native" is really a great total argument. Why not ask Admin to create an R&D sheet generator, or a native R&D hotkey tool? Navigating those rules have led to illegalities before!

I can see the merit in requesting a new api call for information - there's a whole thread for that actually, top pin in Techncial. We've been trying to get minor times in a call for one.

But a whole special interface for, as you said, 30 regions at most? Hah.

[Refuge Isle]

But a whole special interface for, as you said, 30 regions at most? Hah.

It's worth noting that the "30 regions at most" account for 12,000 players -- half of all World Assembly nations.

[Dawn Denac]

But a whole special interface for, as you said, 30 regions at most? Hah.

It's worth noting that the "30 regions at most" account for 12,000 players -- half of all World Assembly nations.

So how do you view the API call Souls pointed out, or a simpler solution than what Sandaoguo is suggesting?

EDIT: Moreso for the fact I want to state I'm not against the idea of something akin to this, but not in the way proposed in OP, curious on inputs from other people.

[Ever-Wandering Souls]

I'm curious where HTML calls even come into things if you're willing to accept day-old info, Glen.

The "slow way" of calling endorsements on every nation (or ideally just every wa nation) in a gcr via the api, and then doing a logical test to see who has not received an endo from a specific person takes tens of minutes and a lot of calls, yes, but is relatively simple and safe - you could even, and I'm confident some people do, run it once daily to build the table and just filter it for whoever wants to check.

But even if Admin doesn't want to do an "endorsement given" call for technical reasons, the ideal solution for a daily status would seem to be the nation dump. Given that the Nations Dump has that info daily (endorsements received) you can easily use that to make a daily list that's 100% safe in as little time as it takes to download the dump and let your logic run, avoiding hundreds of API calls. Sure, the logic to collect "endorsements received" by every WA nation in your region and backtrack that to who any specific person hasn't endorsed yet takes a little work, but the tools are already there in a way that doesn't burden the site.

What's wrong with using the dumps for a once daily cross endorsement run? Why all this talk of the HTML being needed? The nation dumps are a safe and low barrier to entry way of scripting this.

Some people use a combination of API calls and dump files. This is a good idea if you want to compare a nation to others in its region. Since regions can contain thousands of nations, it's not practical to request data on all a nation's neighbors at once via the API. Instead, try building region-wide stats (e.g. total region population) using the most recent daily dump, and only drawing on the API for up-to-the-second information on the nation in question.

[United Calanworie]

I'm curious where HTML calls even come into things if you're willing to accept day-old info, Glen.

The "slow way" of calling endorsements on every nation (or ideally just every wa nation) in a gcr via the api, and then doing a logical test to see who has not received an endo from a specific person takes tens of minutes and a lot of calls, yes, but is relatively simple and safe - you could even, and I'm confident some people do, run it once daily to build the table and just filter it for whoever wants to check.

But even if Admin doesn't want to do an "endorsement given" call for technical reasons, the ideal solution for a daily status would seem to be the nation dump. Given that the Nations Dump has that info daily (endorsements received) you can easily use that to make a daily list that's 100% safe in as little time as it takes to download the dump and let your logic run, avoiding hundreds of API calls. Sure, the logic to collect "endorsements received" by every WA nation in your region and backtrack that to who any specific person hasn't endorsed yet takes a little work, but the tools are already there in a way that doesn't burden the site.

What's wrong with using the dumps for a once daily cross endorsement run? Why all this talk of the HTML being needed? The nation dumps are a safe and low barrier to entry way of scripting this.

Some people use a combination of API calls and dump files. This is a good idea if you want to compare a nation to others in its region. Since regions can contain thousands of nations, it's not practical to request data on all a nation's neighbors at once via the API. Instead, try building region-wide stats (e.g. total region population) using the most recent daily dump, and only drawing on the API for up-to-the-second information on the nation in question.

I personally use a combination of the region dump and nation dump in order to produce swapfiles. If you know the region for your target ahead of time, you can search through the dump, and pull a list of nations from that region. Then use that list to filter the nation dump, checking the WA status of each of those nations, and pulling the nations endorsing them. Check that list to see if your target is endorsing them, if not, place in another list. It's pretty fast, and if you decompress the dumps and write them to a database, it's even very simple to do with a few lines of SQL.

[Lord Dominator]

Alternatively, why not just manually cross endorse people without scripts? At most you'd have to open a few extra nation links, click endorse and done. A site-coded version of this just sounds like a lot of work for something that is easily doable without scripts or automating it.

There are more than 30 regions with over a hundred WA members, and nine with over five hundred. Endotarting in those regions is more than a few clicks, and constantly re-checking nations that you have missed, as well as new nations that have arrived or WA'd up, is not feasible without scripts.

I tarted the entirety of TNP last year when running for VD solely by using the WA endo rankings initially and a set of filters on the activity feeds to keep up. Whenever I used TNP’s native tool (which updates daily I believe) I inevitably found I already had everyone - or at most that I missed a single person or two in about a week. Tarting in general and to keep up are absolutely possible without scripts (if potentially inconvenient) solely on site tools provided you do it at least weekly (since that’s roughly when the activity feed stuff expires). Even so, once-daily tools that already exist work perfectly well for the catch-up if you don’t keep up.

All together, I don’t really see the need for the site to include features that are already largely possessed by the regions that might need them.

[Sandaoguo]

I'm curious where HTML calls even come into things if you're willing to accept day-old info, Glen.

Archer is open-source and 100% client side, so you can view the source code directly in your browser if you'd like. The GitHub repo is here: https://github.com/hierocles/archer2 The data it's pulling comes from a custom API called Trove that's essentially an abstraction layer of the daily dumps plus a custom call for endo lists: https://github.com/hierocles/trove

Archer is targeted towards the general population, rather than power-users. So I don't bother with more complicated ways to ensuring an up-to-date endorsements list. The audience is the kind of player who endotarts a couple times a week, rather than every day or multiple times a day. The downside is that isn't not always accurate and has a higher fail rate than is ideal.

To generate a more accurate list, endorsement scripts can employ a bot that trawls happenings pages or calls the World API, recording endorsement data and then updating the backend database with the daily dump being used as a baseline. If the tool is only being used by a small number of people, you can use on-demand API calls to update the baseline entries whenever that person uses the tool. Rate-limiting makes that more difficult once the tool is being used by more than a handful of people.

The "slow way" of calling endorsements on every nation (or ideally just every wa nation) in a gcr via the api, and then doing a logical test to see who has not received an endo from a specific person takes tens of minutes and a lot of calls, yes, but is relatively simple and safe - you could even, and I'm confident some people do, run it once daily to build the table and just filter it for whoever wants to check.

This is true if you make one tool per region, instead of a tool for general public consumption. Your estimation for how long it would take to run individual API calls to get this data is a little off, though. There's no "WA members" shard for the Regions API. So you have to run 1 call to get a list of all nations in the region, 1 API call to the WA API to cross-reference regional membership, and then 1 call per member to get their endorsements. For a region the size of TSP, you're looking probably at anywhere from 1.5-2 hours of non-stop API calls per day just to get the baseline data. And even then, that's still not going to provide on-the-fly accuracy, which requires on-demand API calls, which you wouldn't be able to do during the "update" period because you're maxing out your rate limit bucket already.

For those on-demand calls, you're looking at a user request taking several minutes (for a region with 1000 WA members, 10 minutes minus whatever efficiencies you can get by recording past uses). And that's only if you max out the rate limit on one single user, since the API rate limits by source, not by end-user.

What's wrong with using the dumps for a once daily cross endorsement run? Why all this talk of the HTML being needed? The nation dumps are a safe and low barrier to entry way of scripting this.

While this is the route I've gone with Archer, mostly because I don't want to write more complicated code for it, it's not actually 100% accurate and never will be. It's accurate only for the slice of time the daily data dumps are generated. Using that method, end users run into many nations they're no longer eligible to endorse. I haven't measured the failure rate, but Archer (and any other similar tool) has a noticeable failure rate, and it's been one of the most common complaints I received from beta testers and those who use the released version. In order to eliminate those failures, you have to monitor the happenings pages or World API.

That's your dev lesson from me for the day. It's not really all that relevant to the feature request I'm making. NS admins have a preference against scripts existing, particularly when they need to employ bots. Instead of complaining about that preference, I'm suggestion to [v] and other dev admins that they add this as a native feature of NS. These types of endorsement tools (targeted towards permanent residents of regions, not R/D endoswappers) exist in almost every major region, third-party devs reinvent them all the time, and they're relatively "costly" in terms of the amount of data being extracted through dumps, API calls, or page scraping. It's really a no-brainer feature. These tools have existed in their current form for at least the past 6 years, and even longer in earlier forms. So whatever Gameplay balancing concerns someone can think of are already moot-- the game has already been "balanced" in response to them, because they've been part of the third-party infrastructure for so long. And because these are legal tools that will continue to exist, whether or not NS implements the feature natively has no impact on game balance.

I tarted the entirety of TNP last year when running for VD solely by using the WA endo rankings initially and a set of filters on the activity feeds to keep up.

This is great and maybe you deserve a medal of recognition for it. But despite your skill here, the third-party tools still exist and will continue to exist.

[Aurum Raider]

These types of endorsement tools (targeted towards permanent residents of regions, not R/D endoswappers) exist in almost every major region, third-party devs reinvent them all the time, and they're relatively "costly" in terms of the amount of data being extracted through dumps, API calls, or page scraping. It's really a no-brainer feature. These tools have existed in their current form for at least the past 6 years, and even longer in earlier forms. So whatever Gameplay balancing concerns someone can think of are already moot-- the game has already been "balanced" in response to them, because they've been part of the third-party infrastructure for so long. And because these are legal tools that will continue to exist, whether or not NS implements the feature natively has no impact on game balance.

My earlier opposition was a result of not fully understanding your proposal. Now that I understand better what you're proposing I think it is genuinely a good idea.

[Lord Dominator]

I tarted the entirety of TNP last year when running for VD solely by using the WA endo rankings initially and a set of filters on the activity feeds to keep up.

This is great and maybe you deserve a medal of recognition for it. But despite your skill here, the third-party tools still exist and will continue to exist.

My point was a refutation of a specific argument that tarting in a large region is impossible without scripts, as can be observed in my going on to discuss other topics from my perspective as a mere tarter. I don’t view anything beyond a hypothetical onsite recreation of the scripts that update daily or so for endorsement comparisons to be worthwhile for that reason - it’s easy enough to tart with one of those daily, and if you’re going multiple times daily the activity feed works well enough on a move+join filter (that is, to catch nations moving to the region and joining WA) to catch the new people.

This base level of functionality already exists as you note (of course, that’s partly also an argument against including it in the site itself) and I don’t think there’s a good justification to get more advanced than this for tools within the game. At most, maybe the ability separate moves in/out of a region on the activity page besides the once-daily, as that would cut off what is in my experience the large majority of the wasted effort on daily tarting.

I make no comment on the API and script stuff, for I know very little about either.

[Sandaoguo]

You've missed the point in my response. Whether you personally see the value or not doesn't matter, the tools exist for a reason. Pointing out that you can do manually what the tools make far easier is a useless observation. The tools exist specifically to not have to do things manually. And since they aren't illegal, they're never going away. The purpose of this thread is to suggest to admins that they can make the third-party tools redundant and obviate any potential script legality issues by taking what the tools do and make that a native feature. Much like how third-party tool development started way back when with auto-TG scripts and page scraping, where admins took that and created the API, TG stamps, templates, etc.

[Lord Dominator]

You've missed the point in my response. Whether you personally see the value or not doesn't matter, the tools exist for a reason. Pointing out that you can do manually what the tools make far easier is a useless observation. The tools exist specifically to not have to do things manually. And since they aren't illegal, they're never going away. The purpose of this thread is to suggest to admins that they can make the third-party tools redundant and obviate any potential script legality issues by taking what the tools do and make that a native feature. Much like how third-party tool development started way back when with auto-TG scripts and page scraping, where admins took that and created the API, TG stamps, templates, etc.

…I’m not strictly disagreeing with building some of it into the site though. I don’t strictly think it’s a priority because legal scripts already exist, I only purely disagree with the high-end “anyone not endorsed in a region at a given time” functionality being site-included. My original post in here may have been a bit negative, but it was originally just to purely point out that manual tarting remains possible (if time-consuming, I’m not clear how much compared to script use) - it may have escaped from me after that.

Edit: That is, I think the once-a-day version (or once an update) would be nice - I added a note about potential activity page improvements to similarly bring that closer to script functionality (and generally improve usage, to be honest) as I understand it exists. I simply don’t think anything updated more regularly than update should be made.

[Kethania]

This is true if you make one tool per region, instead of a tool for general public consumption. Your estimation for how long it would take to run individual API calls to get this data is a little off, though. There's no "WA members" shard for the Regions API. So you have to run 1 call to get a list of all nations in the region, 1 API call to the WA API to cross-reference regional membership, and then 1 call per member to get their endorsements. For a region the size of TSP, you're looking probably at anywhere from 1.5-2 hours of non-stop API calls per day just to get the baseline data. And even then, that's still not going to provide on-the-fly accuracy, which requires on-demand API calls, which you wouldn't be able to do during the "update" period because you're maxing out your rate limit bucket already.

For those on-demand calls, you're looking at a user request taking several minutes (for a region with 1000 WA members, 10 minutes minus whatever efficiencies you can get by recording past uses). And that's only if you max out the rate limit on one single user, since the API rate limits by source, not by end-user.

It is, in fact, inefficient to do a thing in an inefficient way.

Consider doing that thing more efficiently. If you already maintain an "abstraction layer" running 24/7, use it to parse new happenings. That's what I do myself. If you launch a script on request without saving any state, consider not doing that download the daily dump and parse the happenings from the last <24 hours. This is not some sort of infeasible task, making it sound that way is disingenuous.

More generally, writing a bot that interfaces with NS in any way is not for the faint of heart. You are using a system that was developed very quickly 20 years ago without much regard for architecture, and then built and built upon, all in a dying language and by a small team of overworked devs. You should expect hardship. That's your dev lesson from me for the day.

I can see the merit in requesting a new api call for information - there's a whole thread for that actually, top pin in Techncial. We've been trying to get minor times in a call for one.

This was already brought up and addressed 3 years ago by [violet]:

At present I have to hit the API with hundreds of requests (1 for WAs, 1 for region, then go through each WA in the region to check if X has endorsed them).

We don't have a server-side method of generating that information quickly enough for the API. It requires hundreds of fetches on this side, too.

You can figure it out from Daily Dumps, though, as Northern Light says.

Given this, all the R/D concerns don't really matter, and this whole line of discussion kind of void. It's no more efficient to implement for the admins server-side than it is for us client-side.

[Ever-Wandering Souls]

This is true if you make one tool per region, instead of a tool for general public consumption. Your estimation for how long it would take to run individual API calls to get this data is a little off, though. There's no "WA members" shard for the Regions API. So you have to run 1 call to get a list of all nations in the region, 1 API call to the WA API to cross-reference regional membership, and then 1 call per member to get their endorsements. For a region the size of TSP, you're looking probably at anywhere from 1.5-2 hours of non-stop API calls per day just to get the baseline data. And even then, that's still not going to provide on-the-fly accuracy, which requires on-demand API calls, which you wouldn't be able to do during the "update" period because you're maxing out your rate limit bucket already.

For those on-demand calls, you're looking at a user request taking several minutes (for a region with 1000 WA members, 10 minutes minus whatever efficiencies you can get by recording past uses). And that's only if you max out the rate limit on one single user, since the API rate limits by source, not by end-user.

It is, in fact, inefficient to do a thing in an inefficient way.

Consider doing that thing more efficiently. If you already maintain an "abstraction layer" running 24/7, use it to parse new happenings. That's what I do myself. If you launch a script on request without saving any state, consider not doing that download the daily dump and parse the happenings from the last <24 hours. This is not some sort of infeasible task, making it sound that way is disingenuous.

More generally, writing a bot that interfaces with NS in any way is not for the faint of heart. You are using a system that was developed very quickly 20 years ago without much regard for architecture, and then built and built upon, all in a dying language and by a small team of overworked devs. You should expect hardship. That's your dev lesson from me for the day.

I can see the merit in requesting a new api call for information - there's a whole thread for that actually, top pin in Techncial. We've been trying to get minor times in a call for one.

This was already brought up and addressed 3 years ago by [violet]:

We don't have a server-side method of generating that information quickly enough for the API. It requires hundreds of fetches on this side, too.

You can figure it out from Daily Dumps, though, as Northern Light says.

Given this, all the R/D concerns don't really matter, and this whole line of discussion kind of void. It's no more efficient to implement for the admins server-side than it is for us client-side.

Yes, that's why I focused on the dumps method. Since Glen said he was fine with once daily.