NATION

PASSWORD

Script: "Reliant" + HTML Script Legality Discussion

Bug reports, general help, ideas for improvements, and questions about how things are meant to work.

Advertisement

Remove ads

User avatar
United Calanworie
Envoy
 
Posts: 219
Founded: Dec 12, 2018
Democratic Socialists

Postby United Calanworie » Thu Jun 16, 2022 10:07 pm

[violet] wrote:So fyi here are what I think are some reasonable concerns people might have about being shifted to the API:
  • Admin might not actually get around to adding the API endpoints needed

Literally this one right here is what I am the most concerned about. There are features going back to 2021 that are things that could really be useful to the community. And they haven't been implemented. Not only have they not been implemented, there's been no indications that they've been seen. I recognize that admin time is limited, but even just a post in the thread saying "we've seen your suggestion" would be a billion times better than the status quo of not knowing. Some of those features might be more complicated (API reading telegrams), others are just reformatting information that already exists/adding a line to the daily data dumps (minor update time shard, nation SPDR score added to dumps). I really don't want to sound like I'm complaining, but... it's inherently frustrating as a 3rd party developer to have to work around these problems and not even know if admin is seeing our posts about them.
Discord: Aav#7564
She/Her/Hers
My Projects:
HYPR: TG API software (GitHub) | Spyglass (forum) (GitHub) (Latest Release)

||||||||||||||||||||
||||||||||||||||||||

User avatar
Zizou
Diplomat
 
Posts: 560
Founded: Aug 23, 2018
Iron Fist Consumerists

Postby Zizou » Thu Jun 16, 2022 10:35 pm

United Calanworie wrote:
[violet] wrote:So fyi here are what I think are some reasonable concerns people might have about being shifted to the API:
  • Admin might not actually get around to adding the API endpoints needed

Literally this one right here is what I am the most concerned about. There are features going back to 2021 that are things that could really be useful to the community. And they haven't been implemented. Not only have they not been implemented, there's been no indications that they've been seen. I recognize that admin time is limited, but even just a post in the thread saying "we've seen your suggestion" would be a billion times better than the status quo of not knowing. Some of those features might be more complicated (API reading telegrams), others are just reformatting information that already exists/adding a line to the daily data dumps (minor update time shard, nation SPDR score added to dumps). I really don't want to sound like I'm complaining, but... it's inherently frustrating as a 3rd party developer to have to work around these problems and not even know if admin is seeing our posts about them.

Hard agree on this. Admin silence and (perhaps perceived) lack of progress on community requested features are absolutely maddening. Without trying to sound too harsh, Admin has developed a tried and true track record of either complete radio silence on our suggestions, or responding positively to certain suggestions and having absolutely no indication of developement progress made on them.

Refuge Isle wrote:To further reiterate my concerns stated previously, A 50/30s rate limit is equally unworkable if the status quo is refreshing 10 times a second looking for that activity. As wins and losses are already bottlenecked by human reaction time, restricting them further by severely limiting how many times you can refresh puts us back at square one where even manually refreshing an HTML reports page a billion times would improve our chances of success.

Assuming the happenings are migrated to WebSockets or SSE like Violet was talking about earlier, hopefully 50/30 shouldn't end up being a problem.

[violet] wrote:
  • People might be outcompeted by fully-automated API bots

This is a bit of a question I had in the back of my mind. I don't think the issue is really "outcompeting" per se, since both Raiderdom and Defenderdom are quite united and both have excellent coders at their disposal. My concern here is that new unrestricted API endpoints for things like moving and joining the WA are going to lead to near complete automation of the R/D game, which is a situation I think myself and a lot of others would wish to avoid.
Last edited by Zizou on Thu Jun 16, 2022 10:36 pm, edited 1 time in total.
Zizou Vytherov-Skollvaldr
LTN in The Black Hawks
Meishu of the former Red Sun Army
Parxland wrote:It might somehow give me STDs through the computer screen with how often you hop between different groups of people.

User avatar
Wallenburg
Postmaster of the Fleet
 
Posts: 22091
Founded: Jan 30, 2015
Liberal Democratic Socialists

Postby Wallenburg » Thu Jun 16, 2022 10:53 pm

Zizou wrote:Hard agree on this. Admin silence and (perhaps perceived) lack of progress on community requested features are absolutely maddening. Without trying to sound too harsh, Admin has developed a tried and true track record of either complete radio silence on our suggestions, or responding positively to certain suggestions and having absolutely no indication of developement progress made on them.

Or deliberately pursuing features that the community positively opposes whilst either ignoring or silencing expression of that opposition. In any case, admin has a bad habit of avoiding engagement with the community they claim to volunteer for, which results in a lack of trust that admin will actually do things that help.
Last edited by Wallenburg on Thu Jun 16, 2022 10:56 pm, edited 1 time in total.
I want to improve.
grestin went through the MKULTRA program and he has more of a free will than wallenburg does - Imperial Idaho
Kiu Ghesik wrote:harris' interpretation of bidenism and subsequent establishment of a bidenist vanguard party to root out malarkey and revisionist elements in society was revisionist in and of itself and should never have been implemented.

King of Snark, Minister of World Assembly Affairs, Arbiter for The East Pacific

User avatar
United Calanworie
Envoy
 
Posts: 219
Founded: Dec 12, 2018
Democratic Socialists

Postby United Calanworie » Thu Jun 16, 2022 10:55 pm

Zizou wrote:
Refuge Isle wrote:To further reiterate my concerns stated previously, A 50/30s rate limit is equally unworkable if the status quo is refreshing 10 times a second looking for that activity. As wins and losses are already bottlenecked by human reaction time, restricting them further by severely limiting how many times you can refresh puts us back at square one where even manually refreshing an HTML reports page a billion times would improve our chances of success.

Assuming the happenings are migrated to WebSockets or SSE like Violet was talking about earlier, hopefully 50/30 shouldn't end up being a problem.

This is basically what I've thought about this as well -- as long as we can rely on the websocket/SSE feed for activity, the API rate limit wouldn't be a huge problem. If, however, maintaining a happenings feed pull costs API requests... oh boy, that will produce a *lot* of problems.

Zizou wrote:
[violet] wrote:
  • People might be outcompeted by fully-automated API bots

This is a bit of a question I had in the back of my mind. I don't think the issue is really "outcompeting" per se, since both Raiderdom and Defenderdom are quite united and both have excellent coders at their disposal. My concern here is that new unrestricted API endpoints for things like moving and joining the WA are going to lead to near complete automation of the R/D game, which is a situation I think myself and a lot of others would wish to avoid.

Complete automation of the R/D game would suck. It really, truly would. There aren't a lot of good ways to enforce lack of automation for these endpoints that don't wind up being in the same situation as the scripting rules, but at the end of the day I think we'd need to avoid any form of region-change endpoint, and we'd need to see WA apps becoming a private command, and perhaps requiring X-Password/X-Autologin authentication instead of X-Pin, so that they're slowed down significantly.
Discord: Aav#7564
She/Her/Hers
My Projects:
HYPR: TG API software (GitHub) | Spyglass (forum) (GitHub) (Latest Release)

||||||||||||||||||||
||||||||||||||||||||

User avatar
Zizou
Diplomat
 
Posts: 560
Founded: Aug 23, 2018
Iron Fist Consumerists

Postby Zizou » Thu Jun 16, 2022 11:57 pm

Wallenburg wrote:
Zizou wrote:Hard agree on this. Admin silence and (perhaps perceived) lack of progress on community requested features are absolutely maddening. Without trying to sound too harsh, Admin has developed a tried and true track record of either complete radio silence on our suggestions, or responding positively to certain suggestions and having absolutely no indication of developement progress made on them.

Or deliberately pursuing features that the community positively opposes whilst either ignoring or silencing expression of that opposition. In any case, admin has a bad habit of avoiding engagement with the community they claim to volunteer for, which results in a lack of trust that admin will actually do things that help.

While I think a lot of what you said is true, I did think on things a bit more, and I'm actually tentatively optimistic about developement of new API features for a few reasons:
  • For one, this is something which must be addressed. Violet has made it pretty clear that the plan for this is a migration off of HTML scripts towards the API. Hence, getting the new API completed is essential for changing the status quo, which is untenable for both the Admins and users.
  • Secondly, it seems like a decent amount of work has already gone into implementing new API features recently. Shards for banners were implemented, and the SSE system that's being built in Node for a new happenings API is quite a monumental step to finally begin moving the API off of ancient Perl CGI scripts.
And while obviously there's a lot more work to be done, this is at least a good start, and I hope Admin continues in this direction.

United Calanworie wrote: Complete automation of the R/D game would suck. It really, truly would. There aren't a lot of good ways to enforce lack of automation for these endpoints that don't wind up being in the same situation as the scripting rules, but at the end of the day I think we'd need to avoid any form of region-change endpoint, and we'd need to see WA apps becoming a private command, and perhaps requiring X-Password/X-Autologin authentication instead of X-Pin, so that they're slowed down significantly.

This really doesn't seem like a good solution either to be honest. I haven't worked with private commands much, so I can't speak for how much that would slow things down (although logically, I'd think any endpoint added that actually changed anything about a nation would have to be a private command, like moving or applying to the WA or whatever else). But the lack of a move endpoint would absolutely kill this. Without a move endpoint, the script would have to open the region page in the (meaning it makes an HTML request), or that the scripts would be made pretty useless, because at that point, all you really have is an alternate version of the happenings feed.
Zizou Vytherov-Skollvaldr
LTN in The Black Hawks
Meishu of the former Red Sun Army
Parxland wrote:It might somehow give me STDs through the computer screen with how often you hop between different groups of people.

User avatar
United Calanworie
Envoy
 
Posts: 219
Founded: Dec 12, 2018
Democratic Socialists

Postby United Calanworie » Fri Jun 17, 2022 12:02 am

Zizou wrote:
United Calanworie wrote: Complete automation of the R/D game would suck. It really, truly would. There aren't a lot of good ways to enforce lack of automation for these endpoints that don't wind up being in the same situation as the scripting rules, but at the end of the day I think we'd need to avoid any form of region-change endpoint, and we'd need to see WA apps becoming a private command, and perhaps requiring X-Password/X-Autologin authentication instead of X-Pin, so that they're slowed down significantly.

This really doesn't seem like a good solution either to be honest. I haven't worked with private commands much, so I can't speak for how much that would slow things down (although logically, I'd think any endpoint added that actually changed anything about a nation would have to be a private command, like moving or applying to the WA or whatever else). But the lack of a move endpoint would absolutely kill this. Without a move endpoint, the script would have to open the region page in the (meaning it makes an HTML request), or that the scripts would be made pretty useless, because at that point, all you really have is an alternate version of the happenings feed.

I mean... just spitballing here, but what if the endpoint returned status 503 during update? Would be functional for prepping puppets and such, but absolutely useless for in-update tactics.
Discord: Aav#7564
She/Her/Hers
My Projects:
HYPR: TG API software (GitHub) | Spyglass (forum) (GitHub) (Latest Release)

||||||||||||||||||||
||||||||||||||||||||

User avatar
Zizou
Diplomat
 
Posts: 560
Founded: Aug 23, 2018
Iron Fist Consumerists

Postby Zizou » Fri Jun 17, 2022 12:12 am

United Calanworie wrote:
Zizou wrote:
This really doesn't seem like a good solution either to be honest. I haven't worked with private commands much, so I can't speak for how much that would slow things down (although logically, I'd think any endpoint added that actually changed anything about a nation would have to be a private command, like moving or applying to the WA or whatever else). But the lack of a move endpoint would absolutely kill this. Without a move endpoint, the script would have to open the region page in the (meaning it makes an HTML request), or that the scripts would be made pretty useless, because at that point, all you really have is an alternate version of the happenings feed.

I mean... just spitballing here, but what if the endpoint returned status 503 during update? Would be functional for prepping puppets and such, but absolutely useless for in-update tactics.

The thing is, these endpoints do need to be usable during update. If a script like, say Breeze for instance (which, as many defenders have expressed, is essentially a necessity for chasing), were to be migrated off of making HTML requests to the API, the move endpoint needs to be functional without being able to be completely automated.

I missed this:
[violet] wrote:I don't think anyone is talking about banning simple keybinds of that kind. There is, of course, a wide and expanding range of browser tools and add-ons, many of which blur the lines between user and program. But what we're talking about today are scripts that craft and dispatch custom HTML requests.

With this in mind, that does seem like band-aid solution, but I suppose it would serve its purpose.
Last edited by Zizou on Fri Jun 17, 2022 12:20 am, edited 2 times in total.
Zizou Vytherov-Skollvaldr
LTN in The Black Hawks
Meishu of the former Red Sun Army
Parxland wrote:It might somehow give me STDs through the computer screen with how often you hop between different groups of people.

User avatar
Esfalsa
Secretary
 
Posts: 26
Founded: Aug 07, 2015
Civil Rights Lovefest

Postby Esfalsa » Fri Jun 17, 2022 8:28 am

I understand concerns about a move command, but I’m not sure I understand, what’s the issue with an API command to apply to the WA?

While we’re on this topic, I also wanted to mention that a lot of what defenders do currently as part of ‘chasing’ can feel… cumbersome. When we add raider nations to our dossier, it’s two HTML requests per nation (to open the nation page and click the ‘add to dossier’ button). When we endorse other defenders, it’s two HTML requests per nation (to open the nation page and click the ‘endorse’ button).

Switching is something both raiders and defenders do and that’s felt even more cumbersome at times — it’s one HTML request to see if the region/puppet has updated (more if the region updates later than predicted by the trigger), one to navigate to page=un, one to click the resign button, one to open the emailed WA admit link, and one to click the ‘confirm’ button.

Of course, API endpoints for all of these would pose risks both in terms of the automation of R/D gameplay and also to game balance, and in all honestly I’m not sold on any ways of working around that right now. But I just wanted to add the flip side to the conversation as well.
Last edited by Esfalsa on Fri Jun 17, 2022 8:42 am, edited 1 time in total.

User avatar
Racoda
Spokesperson
 
Posts: 169
Founded: Aug 12, 2014
New York Times Democracy

Postby Racoda » Sun Jun 19, 2022 4:05 pm

[violet] wrote:
SherpDaWerp wrote:If there was a bit more specific wording employed (i.e. less "ban html scripts" and more "ban html request-making scripts") or some Official Wording about what's actually being considered, that would quell a lot of fears in the Cards community. It's a bit like putting the cart before the horse, I know, drafting a wording for a rule that you're not even convinced you need yet, but if there was a specific "admin do not intend to ban your QoL tool" post to point at, there'd be a few less Cards voices against this change.

Well maybe "ban" is the wrong word, since I'm actually talking about a migration, whereby scripts that currently send requests to https://www.nationstates.net would send them to api.nationstates.net instead. (It's a bit more involved than that, admittedly, but not a whole lot.) I don't want to ban any tools -- I want to change where they're sending traffic. And if they're not sending any traffic to the HTML site, they're not my concern.


Thank you for this response, this clarifies a lot about what the "HTML script ban" is targeting.

There is a distinction to be made between what I will call passive, reactive and active scripts:
Passive scripts are scripts that enhance the UI/UX of NationStates, without sending HTML requests and without adding additional buttons to the vanilla site that the user can click to send requests. This includes, amongst others, pure CSS modifications, but also extends to stuff that can't be implemented in pure CSS. As I'm a cards scripter, here's a few examples of the stuff I've written for cards that is essentially just UI enhancement:
Better Card Page Titles: modifies the titles of card page so that they're easily searchable in browser history. Technical request thread
(See title in top left)Image
Cards Icon: just adds a quick link to the deck and the market to the top banner (next to Settings)
Image
Guild/Society Highlighter: adds an icon next to nations that are part of card guilds or societies, usually to avoid heisting between fellow members
Image
Main nation displayer: Adds a link to the "main" nation of a puppet, so that TGs can be sent to the puppetmaster directly instead of the puppet (where they're likely to get ignored).
Image
Alternate Auction Layout: Rearranges the auction page to avoid scrolling on wider screens, also collapses multiple identical trade offers into one (see red circle)
Image


My point being, all of these are clearly UI/UX enhancements and not "bots", but they wouldn't be able to do this without modifying the HTML page. Furthermore, they're not sending any requests to NS, which is why I call them "passive".

Then there's the "reactive" scripts, which send requests to, or facilitate sending requests to NS. They however don't craft those requests themselves, but rely on NS buttons or links. Which is what keybindings fall into. The one most used in cards is probable this one. For obvious reasons, there isn't really a screenshot to include. Other examples of this are the farming keybinds, where the issue page is minimized to only the answering buttons and one just needs to press enter to answer the issue.
Image


Finally, there's the scripts that craft their own requests to the HTML site, be it through buttons or keybinds. The only one I can think of is my unfinished overhaul to the deck page, which adds "junk" buttons in the rows of the value_deck page and allows to filter cards based on different conditions. This tool does craft it's own requests, they are however always a consequence of user input.
Image

This script in particular is impossible to port to the API: there's no "junk card" private command (and I doubt one will ever be added). Despite crafting its own requests, it remains a QoL script rather than a "bot". Ironically, it is also much slower (due to simultaneity) than just using a junking keybind on the normal deck page. This script would most likely be banned under any new rules, and is impossible to migrate to the API.




All that being said, what I'd love to see from the API is:
  • A move to JSON. XML is passé and cumbersome, and in some cases the API returns malformed XML (!). An example that I've had to deal with is the XML returned for issue 610, where Dàguó is not properly encoded in XML. Some lax parsers might let it pass, but I haven't found any such lax parsers for python for example. This wrongly-encoded-XML problem also affects the cards dumps, which I've had to re-encode for them to be properly parsed.
  • A possibility to authenticate differently than through headers (ex. a POST request with a password field). For WebApps, a CORS request is sent first if the app tries to send a request containing non-standard headers (like X-Password). However, the app is not notified if/when a CORS request is sent and those count towards the rate limit. This has been a known issue for 5 years.
RCES - Farming and QoL cards scripts

User avatar
Eluvatar
Site Admin
 
Posts: 2525
Founded: Mar 31, 2006
New York Times Democracy

Postby Eluvatar » Sun Jun 19, 2022 6:12 pm

I think you might be underestimating how much we may add API-side.
To Serve and Protect: UDL

Eluvatar - Taijitu member

User avatar
Syberis
Diplomat
 
Posts: 643
Founded: Jan 21, 2016
Moralistic Democracy

Postby Syberis » Sun Jun 19, 2022 9:38 pm

Eluvatar wrote:I think you might be underestimating how much we may add API-side.


Then perhaps someone should tell us what you may add API-side. After all, it's been a year now since creation of Development Managers, which we were led to believe the whole point of that was to form a pipeline in addition to separation of tasks.

I'm not asking for full-blown dev diaries but perhaps more than vagueposting about what the team might do will help people get some faith back in what's going on. I'm not asking you guys to promise things but at least give us a glimpse into things. Perhaps giving us literally any sort of communication beyond this that anything is happening will prevent underestimation when we have little reason not to, since we've been waiting on a lot of "things we may add" for a very long time.

Like I'm sorry to be so hot in this thread but you guys really aren't giving us anything of substance and then are consistently surprised when we're frustrated about it. There's a fundamental breakdown of communication here, I think.
I've finally found what I was looking for
A place where I can be without remorse
Because I am a stranger who has found
An even stranger war

Zaolat wrote:WHO THE F*** IS SYBERIS

User avatar
Trotterdam
Powerbroker
 
Posts: 9785
Founded: Jan 12, 2012
Left-Leaning College State

Postby Trotterdam » Sun Jun 19, 2022 10:19 pm

Racoda wrote:A move to JSON. XML is passé and cumbersome, and in some cases the API returns malformed XML (!).
Both JSON and XML include a bunch of complicated formatting metadata that you ideally need a dedicated parser to make sense of, rather than just dumping the data in an as simple and easy-to-parse format as possible. If I'm going to need to deal with that anyway, I'd rather stick with the format I've already implemented, rather than having to scramble to rewrite every NationStates script ever made to adapt to the suddenly-changed format. I suppose it would be possible to include a toggle parameter to allow individual API users to select whether they want to use the JSON or XML interface, but that would require twice as much code that needs to be kept up-to-date and debugged.

I have noticed the XML interface's poor handling of non-ASCII characters and already reported it long ago, along with two subsequent reminders, but it never got any sort of response, and I can confirm the problem is still there. (Note that it's not exactly malformed XML - it's legal XML that a parser will accept, but just displays a result slightly different than what's intended. There's no rule saying that you can't encode a JSON string in XML, or vice versa, if you really want, although there's no good reason to 99.999% of the time. Is your example different?)

User avatar
United Calanworie
Envoy
 
Posts: 219
Founded: Dec 12, 2018
Democratic Socialists

Postby United Calanworie » Mon Jun 20, 2022 12:39 am

Trotterdam wrote:
Racoda wrote:A move to JSON. XML is passé and cumbersome, and in some cases the API returns malformed XML (!).
If I'm going to need to deal with that anyway, I'd rather stick with the format I've already implemented, rather than having to scramble to rewrite every NationStates script ever made to adapt to the suddenly-changed format. I suppose it would be possible to include a toggle parameter to allow individual API users to select whether they want to use the JSON or XML interface, but that would require twice as much code that needs to be kept up-to-date and debugged.

I mean realistically that option already exists, see https://www.nationstates.net/pages/api.html#versions for instructions on how to version your queries to only hit one version of the API.
Discord: Aav#7564
She/Her/Hers
My Projects:
HYPR: TG API software (GitHub) | Spyglass (forum) (GitHub) (Latest Release)

||||||||||||||||||||
||||||||||||||||||||

User avatar
Sedgistan
Senior Issues Moderator
 
Posts: 31926
Founded: Oct 20, 2006
Anarchy

Postby Sedgistan » Mon Jun 20, 2022 2:28 am

Syberis wrote:
Eluvatar wrote:I think you might be underestimating how much we may add API-side.


Then perhaps someone should tell us what you may add API-side. After all, it's been a year now since creation of Development Managers, which we were led to believe the whole point of that was to form a pipeline in addition to separation of tasks.

The thread topic is now effectively "what do people want from the API to replicate what they could do with html site scripts" - so the question is really for players to indicate what they want added. I read Elu's post as a suggestion to be more ambitious with requests.

On DMs, none of the roles really cover the API - that's more of an admin area of responsibility, although we'll give feedback on potential changes that impact on the areas we cover.

User avatar
Trotterdam
Powerbroker
 
Posts: 9785
Founded: Jan 12, 2012
Left-Leaning College State

Postby Trotterdam » Mon Jun 20, 2022 2:59 am

United Calanworie wrote:I mean realistically that option already exists, see https://www.nationstates.net/pages/api.html#versions for instructions on how to version your queries to only hit one version of the API.
Older versions do eventually get retired. And it would be a problem if new shards get added afterwards and those new shards can only be accessed using the new interface instead of using the same format as the old one.

User avatar
Sweeze
Spokesperson
 
Posts: 158
Founded: Oct 21, 2018
Inoffensive Centrist Democracy

Postby Sweeze » Mon Jun 20, 2022 8:21 am

Sedgistan wrote:The thread topic is now effectively "what do people want from the API to replicate what they could do with html site scripts" - so the question is really for players to indicate what they want added. I read Elu's post as a suggestion to be more ambitious with requests.


if this is the case, here's what i would need in the API for shine (swarm's successor) to have feature parity with the current version that is forced to use the html site:
for prepping puppets for an update:
applying to the WA
moving to a specified region

for tagging/detagging regions after update:
changing WFE's of a region
closing embassies of a region
sending embassy from one region to another
uploading and changing regional flag+banners
mass accepting embassies on one region

for making a uniform puppet set:
changing all customizable nation fields (pretitle, demonym, national capital, religion...)
setting custom nation flag
uploading a national banner and disabling all other national banners

misc:
reviving nations
wouldn't be required for feature parity but holy *fuck* json or toml or some other sane format thats not xml would make life so much more convenient
Last edited by Sweeze on Mon Jun 20, 2022 12:44 pm, edited 3 times in total.
| lily supreme command | the mt army third in command | dev of swarm and feather |
[6:38 PM] Chingis: ... the Tom Brady of R/D
5417+ times tag/detag delegate, 5945+ regions hit, only person to become delegate of 200+ regions in an update (multiple times)
she/they

User avatar
9003
Diplomat
 
Posts: 539
Founded: Oct 25, 2012
Father Knows Best State

Postby 9003 » Mon Jun 20, 2022 8:44 am

To avoid any competitive edge from moving regions you could add a cool down or more effective but more server labor Intense a delay on moving meaning that you send an api command and than 3 seconds later it moves you. This would still allow for scripts to prep puppets quickly and easily with out giving an advantage during update as variance will act up in a 3 second window
proud member of PETZ people for the Ethical Treatment of Zombies

Active member of The cards market place discord

User avatar
[violet]
Site Admin
 
Posts: 15663
Founded: Antiquity

Postby [violet] » Mon Jun 20, 2022 11:55 pm

Racoda wrote:There is a distinction to be made between what I will call passive, reactive and active scripts:

Thanks for those pics -- those tools are really fantastic.

I agree everything you have under "Passive" is outside the scope of this discussion -- they are excellent QoL tools but no requests are being sent to the HTML site. So they're already fine. And "active" scripts should migrate to the API.

"Reactive" contains a bit of a gamut. In the example you link, there are basic key remaps such as "r" for reloading the page, just like you pressed F5. That's a simple navigation shortcut that would make no sense to send to the API. The same goes for things like "Press 'i' to go to your Issues page." However, other elements of that tool are more involved, including some keybinds that issue commands (press "g" to gift a card to your puppet) and a custom puppet builder that's over 200 lines long and generates its own form.

I think the navigation shortcuts are fine, as are keybinds that click things that already exist on the page. But once the tool is crafting its own, new requests (including generating its own forms), or building command-issuing UI elements based on what it read from the page -- i.e. not just acting blindly but having a hand in both sides of the request, both reading and sending -- then those requests should go via the API.

It's hard to delineate this area cleanly, but that seems reasonable to me.

Racoda wrote:This script in particular is impossible to port to the API: there's no "junk card" private command (and I doubt one will ever be added).

Why shouldn't one be added -- because you don't think we should permit this action to be fully automated?

Racoda wrote:A move to JSON. XML is passé and cumbersome, and in some cases the API returns malformed XML (!).

Yes, I agree. I won't promise it now, because I want to keep an achievable minimum viable product, but I do think it's unreasonable not to serve JSON in 2022. Our XML parser is also non-standard, which is why it produces bogus output sometimes. Although the specific bug you mention is probably related to our character encoding rather than the parser.

Racoda wrote:A possibility to authenticate differently than through headers (ex. a POST request with a password field). For WebApps, a CORS request is sent first if the app tries to send a request containing non-standard headers (like X-Password). However, the app is not notified if/when a CORS request is sent and those count towards the rate limit.

I've been reluctant to mess with CORS because I'm not very familiar with it and it's full of security issues. But yes, I think this is very possible, as is addressing the issue you link. You would have to provide me with specific instructions on what you want, like you're talking to a five year old who isn't very familiar with CORS.

User avatar
Trotterdam
Powerbroker
 
Posts: 9785
Founded: Jan 12, 2012
Left-Leaning College State

Postby Trotterdam » Tue Jun 21, 2022 12:40 am

[violet] wrote:Although the specific bug you mention is probably related to our character encoding rather than the parser.
It's not a character encoding issue as such - both "\u00AE" (C/JSON/etc.) and "®" (HTML/XML/etc.) are, in their relevant contexts, correct character references that explicitly reference Unicode, as opposed to some other character encoding such as ISO 8859-1 (which would just be "\xAE" in C/JSON/etc., and cannot be directly specified in XML at all except when the entire page is in ISO 8859-1 encoding and you're just serving the raw binary code). It's just that at some point, some part of the code appears to go "hey, this is a Unicode character and I should ASCIIfy it" before the XML constructor appears to ever see it, so it doesn't know that what it's writing will eventually be used for XML and it picks the wrong type of ASCIIfication. I'm guessing whatever piece of code does this is also likely to mangle backslashes as "\" -> "\\", but I can't test this because backslashes are forbidden in custom fields anyway, and they don't seem to appear anywhere in canon.

User avatar
SherpDaWerp
Ambassador
 
Posts: 1376
Founded: Mar 02, 2016
Libertarian Police State

Postby SherpDaWerp » Tue Jun 21, 2022 2:24 am

[violet] wrote:I've been reluctant to mess with CORS because I'm not very familiar with it and it's full of security issues. But yes, I think this is very possible, as is addressing the issue you link. You would have to provide me with specific instructions on what you want, like you're talking to a five year old who isn't very familiar with CORS.

On the CORS topic; while I agree with Racoda's request, what would really help is to exempt OPTIONS requests from the ratelimit, as mentioned here by Elu. That just (well, "just" as much as anything on this website) involves detecting the request type before any processing happens and sending only the specific, potentially hard-coded CORS headers (Access-Control-Allow-Origin, Access-Control-Allow-Methods, Access-Control-Allow-Headers, Access-Control-Max-Age) rather than a full API response (and note that user-agent isn't a special header anymore so you should make sure its on the allowed headers list too).

Racoda's suggestion to optionally add authentication to the body of a POST request would be nice, as it could give us a work-around for the CORS issue and would give a much more web-standard system for authentication, but exempting OPTIONS altogether would be better purely in terms of stopping the CORS problem.


In terms of a laundry list of what would need to be moved off the HTML site in order to not impact Cards scripts, you're mostly just looking at
  • collection management (add/remove cards)
  • puppet creation/management (i.e. changing fields, nation flags, banners, puppet creation, moving)
  • ...and the big one, issues, which no-one wants to see become automated, so this one is less a "new api addition" and more a "new way to make cards issues-less", which is a whole thread worth of discussion in and of itself
Most other Cards scripts that I'm aware of are more on the line of passive/reactive in terms of either adding info to card pages from the API or from player-held servers, or keybinding and queueing cards to process.
Last edited by SherpDaWerp on Tue Jun 21, 2022 2:31 am, edited 1 time in total.

User avatar
Haganham
Diplomat
 
Posts: 936
Founded: Aug 17, 2021
Father Knows Best State

Postby Haganham » Tue Jun 21, 2022 5:09 am

Just kidding. Web3 isn't real.

How can max say this when we were minting NFTs in 2018 :roll:
Remember kids, if it doesn't have private property rights, competitive markets and voluntary exchange then it's just sparkling feudalism.
TITO Tactial Officer
Assistant WA secretary: 10000 Islands, TEP
Praefectus Praetorio, Caesar: Oatland
Cartographer: Forest

User avatar
Roavin
Ambassador
 
Posts: 1215
Founded: Apr 07, 2016
Scandinavian Liberal Paradise

Postby Roavin » Tue Jun 21, 2022 12:31 pm

To add onto the JSON argument: That transition can also be used as an opportunity to represent data more idiomatically, i.e. having something like
Code: Select all
"endorsements": [
   "testlandia",
   "maxtopia"
],

rather than
Code: Select all
<ENDORSEMENTS>testlandia:maxtopia</ENDORSEMENTS>

... which requires additional parsing. Not difficult parsing, but additional parsing nonetheless — it's better to have the API endpoint do the nominal transformation.
Roavinur face from the South Pacific

"Not to mention TSP are some democracy wankers
and democracy wanking is peak defender
" — Tim

NS Coders Discord | I am a LOLcat | Former First Warden of TGW | aka Curious Observations

User avatar
Racoda
Spokesperson
 
Posts: 169
Founded: Aug 12, 2014
New York Times Democracy

Postby Racoda » Wed Jun 22, 2022 7:57 pm

[violet] wrote:"Reactive" contains a bit of a gamut. In the example you link, there are basic key remaps such as "r" for reloading the page, just like you pressed F5. That's a simple navigation shortcut that would make no sense to send to the API. The same goes for things like "Press 'i' to go to your Issues page." However, other elements of that tool are more involved, including some keybinds that issue commands (press "g" to gift a card to your puppet) and a custom puppet builder that's over 200 lines long and generates its own form.

Yes, that's an oversight on my part -- I forgot that script includes a puppet creator. That is obviously in the "active" and not "reactive" category.




[violet] wrote:
Racoda wrote:This script in particular is impossible to port to the API: there's no "junk card" private command (and I doubt one will ever be added).

Why shouldn't one be added -- because you don't think we should permit this action to be fully automated?

SherpDaWerp wrote:
  • ...and the big one, issues, which no-one wants to see become automated, so this one is less a "new api addition" and more a "new way to make cards issues-less", which is a whole thread worth of discussion in and of itself

Mainly this, having fully automated cards farming (fully-automated issue answering as well as fully-automated junking) is a bad idea. Automated junking through HTML scripts is currently forbidden, and if there's going to be an API to junk cards, I think similar restrictions should be imposed (maybe a mandatory userclick/useraction parameter that's checked server-side?).




[violet] wrote:I've been reluctant to mess with CORS because I'm not very familiar with it and it's full of security issues. But yes, I think this is very possible, as is addressing the issue you link. You would have to provide me with specific instructions on what you want, like you're talking to a five year old who isn't very familiar with CORS.

To be honest, I have no experience with CORS -- hence the suggestion to authenticate differently than in headers. I've had a discussion with Sherp after posting and I agree that exempting OPTIONS from the rate-limit (and preventing it from sending an API response) is the best solution, especially since a CORS request will be needed anyway to authorize sending a custom user-agent header. Currently the API returns:
Code: Select all
access-control-allow-headers: X-Pin, X-Password, X-Autologin

which should also include user-agent to enable WebApps to set a user-agent. Note however that there is a nearly 7-year old chromium bug that prevents setting the user-agent on that browser. The one NS JavaScript library I know of goes around this by identifying itself in the URL -- that might remain the only option as long as chromium is bugged.
RCES - Farming and QoL cards scripts

User avatar
Flanderlion
Minister
 
Posts: 2077
Founded: Nov 25, 2013
Psychotic Dictatorship

Postby Flanderlion » Thu Jun 23, 2022 2:59 am

If we're saying what we want from API, sending API TGs using stamps, nations a nation is endorsing list (ridiculous that a nation would not know IC who it says is good), ability to create TG templates by API, and larger dumps of censuses (can just be refreshed when the daily census chooses it) rather than having to do Valentine esque style of just a gazillion requests. Also ability to request in JSON (but keep XML as default just so old scripts don't have to do anything). Also view contents of TGs.

For others, moving/endorsing/etc. all automatic could be done, just not sure if it's really desirable to automate everything. Could require a manual login within 24 hours (or 7 days) for the endo to count?
As always, I'm representing myself.
Information
Wishlist

User avatar
Refuge Isle
Ambassador
 
Posts: 1079
Founded: Dec 14, 2018
Left-wing Utopia

Postby Refuge Isle » Thu Jun 23, 2022 10:36 am

Flanderlion wrote:For others, moving/endorsing/etc. all automatic could be done, just not sure if it's really desirable to automate everything.

Nobody in r/d is keen to see our game reduced to automated scripts. There is no point of it without the human element -- skill, reaction time, dedication.

It would also be kinda wonk if your region got invaded and destroyed by a bot operating independently.

Ikania wrote:My brother in Christ, please stop posting.

PreviousNext

Advertisement

Remove ads

Return to Technical

Who is online

Users browsing this forum: No registered users

Advertisement

Remove ads