NATION

PASSWORD

Script: "Reliant" + HTML Script Legality Discussion

Bug reports, general help, ideas for improvements, and questions about how things are meant to work.

Advertisement

Remove ads

User avatar
United Calanworie
Technical Moderator
 
Posts: 3755
Founded: Dec 12, 2018
Democratic Socialists

Postby United Calanworie » Thu Jun 16, 2022 10:07 pm

[violet] wrote:So fyi here are what I think are some reasonable concerns people might have about being shifted to the API:
  • Admin might not actually get around to adding the API endpoints needed

Literally this one right here is what I am the most concerned about. There are features going back to 2021 that are things that could really be useful to the community. And they haven't been implemented. Not only have they not been implemented, there's been no indications that they've been seen. I recognize that admin time is limited, but even just a post in the thread saying "we've seen your suggestion" would be a billion times better than the status quo of not knowing. Some of those features might be more complicated (API reading telegrams), others are just reformatting information that already exists/adding a line to the daily data dumps (minor update time shard, nation SPDR score added to dumps). I really don't want to sound like I'm complaining, but... it's inherently frustrating as a 3rd party developer to have to work around these problems and not even know if admin is seeing our posts about them.
Trans rights are human rights.
||||||||||||||||||||
Discord: Aav#7546 @queerlyfe
She/Her/Hers
My telegrams are not for Moderation enquiries, those belong in a GHR. Feel free to reach out if you want to just chat.

User avatar
Zizou
Diplomat
 
Posts: 564
Founded: Aug 23, 2018
Ex-Nation

Postby Zizou » Thu Jun 16, 2022 10:35 pm

United Calanworie wrote:
[violet] wrote:So fyi here are what I think are some reasonable concerns people might have about being shifted to the API:
  • Admin might not actually get around to adding the API endpoints needed

Literally this one right here is what I am the most concerned about. There are features going back to 2021 that are things that could really be useful to the community. And they haven't been implemented. Not only have they not been implemented, there's been no indications that they've been seen. I recognize that admin time is limited, but even just a post in the thread saying "we've seen your suggestion" would be a billion times better than the status quo of not knowing. Some of those features might be more complicated (API reading telegrams), others are just reformatting information that already exists/adding a line to the daily data dumps (minor update time shard, nation SPDR score added to dumps). I really don't want to sound like I'm complaining, but... it's inherently frustrating as a 3rd party developer to have to work around these problems and not even know if admin is seeing our posts about them.

Hard agree on this. Admin silence and (perhaps perceived) lack of progress on community requested features are absolutely maddening. Without trying to sound too harsh, Admin has developed a tried and true track record of either complete radio silence on our suggestions, or responding positively to certain suggestions and having absolutely no indication of developement progress made on them.

Refuge Isle wrote:To further reiterate my concerns stated previously, A 50/30s rate limit is equally unworkable if the status quo is refreshing 10 times a second looking for that activity. As wins and losses are already bottlenecked by human reaction time, restricting them further by severely limiting how many times you can refresh puts us back at square one where even manually refreshing an HTML reports page a billion times would improve our chances of success.

Assuming the happenings are migrated to WebSockets or SSE like Violet was talking about earlier, hopefully 50/30 shouldn't end up being a problem.

[violet] wrote:
  • People might be outcompeted by fully-automated API bots

This is a bit of a question I had in the back of my mind. I don't think the issue is really "outcompeting" per se, since both Raiderdom and Defenderdom are quite united and both have excellent coders at their disposal. My concern here is that new unrestricted API endpoints for things like moving and joining the WA are going to lead to near complete automation of the R/D game, which is a situation I think myself and a lot of others would wish to avoid.
Last edited by Zizou on Thu Jun 16, 2022 10:36 pm, edited 1 time in total.
Zizou Vytherov-Skollvaldr
LTN in The Black Hawks
Meishu of the former Red Sun Army
Parxland wrote:It might somehow give me STDs through the computer screen with how often you hop between different groups of people.

User avatar
Wallenburg
Postmaster of the Fleet
 
Posts: 22866
Founded: Jan 30, 2015
Democratic Socialists

Postby Wallenburg » Thu Jun 16, 2022 10:53 pm

Zizou wrote:Hard agree on this. Admin silence and (perhaps perceived) lack of progress on community requested features are absolutely maddening. Without trying to sound too harsh, Admin has developed a tried and true track record of either complete radio silence on our suggestions, or responding positively to certain suggestions and having absolutely no indication of developement progress made on them.

Or deliberately pursuing features that the community positively opposes whilst either ignoring or silencing expression of that opposition. In any case, admin has a bad habit of avoiding engagement with the community they claim to volunteer for, which results in a lack of trust that admin will actually do things that help.
Last edited by Wallenburg on Thu Jun 16, 2022 10:56 pm, edited 1 time in total.
While she had no regrets about throwing the lever to douse her husband's mistress in molten gold, Blanche did feel a pang of conscience for the innocent bystanders whose proximity had caused them to suffer gilt by association.

King of Snark, Real Piece of Work, Metabolizer of Oxygen, Old Man from The East Pacific, by the Malevolence of Her Infinite Terribleness Catherine Gratwick the Sole and True Claimant to the Bears Armed Vacancy, Protector of the Realm

User avatar
United Calanworie
Technical Moderator
 
Posts: 3755
Founded: Dec 12, 2018
Democratic Socialists

Postby United Calanworie » Thu Jun 16, 2022 10:55 pm

Zizou wrote:
Refuge Isle wrote:To further reiterate my concerns stated previously, A 50/30s rate limit is equally unworkable if the status quo is refreshing 10 times a second looking for that activity. As wins and losses are already bottlenecked by human reaction time, restricting them further by severely limiting how many times you can refresh puts us back at square one where even manually refreshing an HTML reports page a billion times would improve our chances of success.

Assuming the happenings are migrated to WebSockets or SSE like Violet was talking about earlier, hopefully 50/30 shouldn't end up being a problem.

This is basically what I've thought about this as well -- as long as we can rely on the websocket/SSE feed for activity, the API rate limit wouldn't be a huge problem. If, however, maintaining a happenings feed pull costs API requests... oh boy, that will produce a *lot* of problems.

Zizou wrote:
[violet] wrote:
  • People might be outcompeted by fully-automated API bots

This is a bit of a question I had in the back of my mind. I don't think the issue is really "outcompeting" per se, since both Raiderdom and Defenderdom are quite united and both have excellent coders at their disposal. My concern here is that new unrestricted API endpoints for things like moving and joining the WA are going to lead to near complete automation of the R/D game, which is a situation I think myself and a lot of others would wish to avoid.

Complete automation of the R/D game would suck. It really, truly would. There aren't a lot of good ways to enforce lack of automation for these endpoints that don't wind up being in the same situation as the scripting rules, but at the end of the day I think we'd need to avoid any form of region-change endpoint, and we'd need to see WA apps becoming a private command, and perhaps requiring X-Password/X-Autologin authentication instead of X-Pin, so that they're slowed down significantly.
Trans rights are human rights.
||||||||||||||||||||
Discord: Aav#7546 @queerlyfe
She/Her/Hers
My telegrams are not for Moderation enquiries, those belong in a GHR. Feel free to reach out if you want to just chat.

User avatar
Zizou
Diplomat
 
Posts: 564
Founded: Aug 23, 2018
Ex-Nation

Postby Zizou » Thu Jun 16, 2022 11:57 pm

Wallenburg wrote:
Zizou wrote:Hard agree on this. Admin silence and (perhaps perceived) lack of progress on community requested features are absolutely maddening. Without trying to sound too harsh, Admin has developed a tried and true track record of either complete radio silence on our suggestions, or responding positively to certain suggestions and having absolutely no indication of developement progress made on them.

Or deliberately pursuing features that the community positively opposes whilst either ignoring or silencing expression of that opposition. In any case, admin has a bad habit of avoiding engagement with the community they claim to volunteer for, which results in a lack of trust that admin will actually do things that help.

While I think a lot of what you said is true, I did think on things a bit more, and I'm actually tentatively optimistic about developement of new API features for a few reasons:
  • For one, this is something which must be addressed. Violet has made it pretty clear that the plan for this is a migration off of HTML scripts towards the API. Hence, getting the new API completed is essential for changing the status quo, which is untenable for both the Admins and users.
  • Secondly, it seems like a decent amount of work has already gone into implementing new API features recently. Shards for banners were implemented, and the SSE system that's being built in Node for a new happenings API is quite a monumental step to finally begin moving the API off of ancient Perl CGI scripts.
And while obviously there's a lot more work to be done, this is at least a good start, and I hope Admin continues in this direction.

United Calanworie wrote: Complete automation of the R/D game would suck. It really, truly would. There aren't a lot of good ways to enforce lack of automation for these endpoints that don't wind up being in the same situation as the scripting rules, but at the end of the day I think we'd need to avoid any form of region-change endpoint, and we'd need to see WA apps becoming a private command, and perhaps requiring X-Password/X-Autologin authentication instead of X-Pin, so that they're slowed down significantly.

This really doesn't seem like a good solution either to be honest. I haven't worked with private commands much, so I can't speak for how much that would slow things down (although logically, I'd think any endpoint added that actually changed anything about a nation would have to be a private command, like moving or applying to the WA or whatever else). But the lack of a move endpoint would absolutely kill this. Without a move endpoint, the script would have to open the region page in the (meaning it makes an HTML request), or that the scripts would be made pretty useless, because at that point, all you really have is an alternate version of the happenings feed.
Zizou Vytherov-Skollvaldr
LTN in The Black Hawks
Meishu of the former Red Sun Army
Parxland wrote:It might somehow give me STDs through the computer screen with how often you hop between different groups of people.

User avatar
United Calanworie
Technical Moderator
 
Posts: 3755
Founded: Dec 12, 2018
Democratic Socialists

Postby United Calanworie » Fri Jun 17, 2022 12:02 am

Zizou wrote:
United Calanworie wrote: Complete automation of the R/D game would suck. It really, truly would. There aren't a lot of good ways to enforce lack of automation for these endpoints that don't wind up being in the same situation as the scripting rules, but at the end of the day I think we'd need to avoid any form of region-change endpoint, and we'd need to see WA apps becoming a private command, and perhaps requiring X-Password/X-Autologin authentication instead of X-Pin, so that they're slowed down significantly.

This really doesn't seem like a good solution either to be honest. I haven't worked with private commands much, so I can't speak for how much that would slow things down (although logically, I'd think any endpoint added that actually changed anything about a nation would have to be a private command, like moving or applying to the WA or whatever else). But the lack of a move endpoint would absolutely kill this. Without a move endpoint, the script would have to open the region page in the (meaning it makes an HTML request), or that the scripts would be made pretty useless, because at that point, all you really have is an alternate version of the happenings feed.

I mean... just spitballing here, but what if the endpoint returned status 503 during update? Would be functional for prepping puppets and such, but absolutely useless for in-update tactics.
Trans rights are human rights.
||||||||||||||||||||
Discord: Aav#7546 @queerlyfe
She/Her/Hers
My telegrams are not for Moderation enquiries, those belong in a GHR. Feel free to reach out if you want to just chat.

User avatar
Zizou
Diplomat
 
Posts: 564
Founded: Aug 23, 2018
Ex-Nation

Postby Zizou » Fri Jun 17, 2022 12:12 am

United Calanworie wrote:
Zizou wrote:
This really doesn't seem like a good solution either to be honest. I haven't worked with private commands much, so I can't speak for how much that would slow things down (although logically, I'd think any endpoint added that actually changed anything about a nation would have to be a private command, like moving or applying to the WA or whatever else). But the lack of a move endpoint would absolutely kill this. Without a move endpoint, the script would have to open the region page in the (meaning it makes an HTML request), or that the scripts would be made pretty useless, because at that point, all you really have is an alternate version of the happenings feed.

I mean... just spitballing here, but what if the endpoint returned status 503 during update? Would be functional for prepping puppets and such, but absolutely useless for in-update tactics.

The thing is, these endpoints do need to be usable during update. If a script like, say Breeze for instance (which, as many defenders have expressed, is essentially a necessity for chasing), were to be migrated off of making HTML requests to the API, the move endpoint needs to be functional without being able to be completely automated.

I missed this:
[violet] wrote:I don't think anyone is talking about banning simple keybinds of that kind. There is, of course, a wide and expanding range of browser tools and add-ons, many of which blur the lines between user and program. But what we're talking about today are scripts that craft and dispatch custom HTML requests.

With this in mind, that does seem like band-aid solution, but I suppose it would serve its purpose.
Last edited by Zizou on Fri Jun 17, 2022 12:20 am, edited 2 times in total.
Zizou Vytherov-Skollvaldr
LTN in The Black Hawks
Meishu of the former Red Sun Army
Parxland wrote:It might somehow give me STDs through the computer screen with how often you hop between different groups of people.

User avatar
Esfalsa
Spokesperson
 
Posts: 132
Founded: Aug 07, 2015
Civil Rights Lovefest

Postby Esfalsa » Fri Jun 17, 2022 8:28 am

I understand concerns about a move command, but I’m not sure I understand, what’s the issue with an API command to apply to the WA?

While we’re on this topic, I also wanted to mention that a lot of what defenders do currently as part of ‘chasing’ can feel… cumbersome. When we add raider nations to our dossier, it’s two HTML requests per nation (to open the nation page and click the ‘add to dossier’ button). When we endorse other defenders, it’s two HTML requests per nation (to open the nation page and click the ‘endorse’ button).

Switching is something both raiders and defenders do and that’s felt even more cumbersome at times — it’s one HTML request to see if the region/puppet has updated (more if the region updates later than predicted by the trigger), one to navigate to page=un, one to click the resign button, one to open the emailed WA admit link, and one to click the ‘confirm’ button.

Of course, API endpoints for all of these would pose risks both in terms of the automation of R/D gameplay and also to game balance, and in all honestly I’m not sold on any ways of working around that right now. But I just wanted to add the flip side to the conversation as well.
Last edited by Esfalsa on Fri Jun 17, 2022 8:42 am, edited 1 time in total.

User avatar
Racoda
Technical Moderator
 
Posts: 579
Founded: Aug 12, 2014
Democratic Socialists

Postby Racoda » Sun Jun 19, 2022 4:05 pm

[violet] wrote:
SherpDaWerp wrote:If there was a bit more specific wording employed (i.e. less "ban html scripts" and more "ban html request-making scripts") or some Official Wording about what's actually being considered, that would quell a lot of fears in the Cards community. It's a bit like putting the cart before the horse, I know, drafting a wording for a rule that you're not even convinced you need yet, but if there was a specific "admin do not intend to ban your QoL tool" post to point at, there'd be a few less Cards voices against this change.

Well maybe "ban" is the wrong word, since I'm actually talking about a migration, whereby scripts that currently send requests to https://www.nationstates.net would send them to api.nationstates.net instead. (It's a bit more involved than that, admittedly, but not a whole lot.) I don't want to ban any tools -- I want to change where they're sending traffic. And if they're not sending any traffic to the HTML site, they're not my concern.


Thank you for this response, this clarifies a lot about what the "HTML script ban" is targeting.

There is a distinction to be made between what I will call passive, reactive and active scripts:
Passive scripts are scripts that enhance the UI/UX of NationStates, without sending HTML requests and without adding additional buttons to the vanilla site that the user can click to send requests. This includes, amongst others, pure CSS modifications, but also extends to stuff that can't be implemented in pure CSS. As I'm a cards scripter, here's a few examples of the stuff I've written for cards that is essentially just UI enhancement:
Better Card Page Titles: modifies the titles of card page so that they're easily searchable in browser history. Technical request thread
(See title in top left)Image
Cards Icon: just adds a quick link to the deck and the market to the top banner (next to Settings)
Image
Guild/Society Highlighter: adds an icon next to nations that are part of card guilds or societies, usually to avoid heisting between fellow members
Image
Main nation displayer: Adds a link to the "main" nation of a puppet, so that TGs can be sent to the puppetmaster directly instead of the puppet (where they're likely to get ignored).
Image
Alternate Auction Layout: Rearranges the auction page to avoid scrolling on wider screens, also collapses multiple identical trade offers into one (see red circle)
Image


My point being, all of these are clearly UI/UX enhancements and not "bots", but they wouldn't be able to do this without modifying the HTML page. Furthermore, they're not sending any requests to NS, which is why I call them "passive".

Then there's the "reactive" scripts, which send requests to, or facilitate sending requests to NS. They however don't craft those requests themselves, but rely on NS buttons or links. Which is what keybindings fall into. The one most used in cards is probable this one. For obvious reasons, there isn't really a screenshot to include. Other examples of this are the farming keybinds, where the issue page is minimized to only the answering buttons and one just needs to press enter to answer the issue.
Image


Finally, there's the scripts that craft their own requests to the HTML site, be it through buttons or keybinds. The only one I can think of is my unfinished overhaul to the deck page, which adds "junk" buttons in the rows of the value_deck page and allows to filter cards based on different conditions. This tool does craft it's own requests, they are however always a consequence of user input.
Image

This script in particular is impossible to port to the API: there's no "junk card" private command (and I doubt one will ever be added). Despite crafting its own requests, it remains a QoL script rather than a "bot". Ironically, it is also much slower (due to simultaneity) than just using a junking keybind on the normal deck page. This script would most likely be banned under any new rules, and is impossible to migrate to the API.




All that being said, what I'd love to see from the API is:
  • A move to JSON. XML is passé and cumbersome, and in some cases the API returns malformed XML (!). An example that I've had to deal with is the XML returned for issue 610, where Dàguó is not properly encoded in XML. Some lax parsers might let it pass, but I haven't found any such lax parsers for python for example. This wrongly-encoded-XML problem also affects the cards dumps, which I've had to re-encode for them to be properly parsed.
  • A possibility to authenticate differently than through headers (ex. a POST request with a password field). For WebApps, a CORS request is sent first if the app tries to send a request containing non-standard headers (like X-Password). However, the app is not notified if/when a CORS request is sent and those count towards the rate limit. This has been a known issue for 5 years.

Acting as a player unless accompagnied by mod action or reddish text
Any pronouns

User avatar
Eluvatar
Director of Technology
 
Posts: 3086
Founded: Mar 31, 2006
New York Times Democracy

Postby Eluvatar » Sun Jun 19, 2022 6:12 pm

I think you might be underestimating how much we may add API-side.
To Serve and Protect: UDL

Eluvatar - Taijitu member

User avatar
Syberis
Diplomat
 
Posts: 689
Founded: Jan 21, 2016
Iron Fist Consumerists

Postby Syberis » Sun Jun 19, 2022 9:38 pm

Eluvatar wrote:I think you might be underestimating how much we may add API-side.


Then perhaps someone should tell us what you may add API-side. After all, it's been a year now since creation of Development Managers, which we were led to believe the whole point of that was to form a pipeline in addition to separation of tasks.

I'm not asking for full-blown dev diaries but perhaps more than vagueposting about what the team might do will help people get some faith back in what's going on. I'm not asking you guys to promise things but at least give us a glimpse into things. Perhaps giving us literally any sort of communication beyond this that anything is happening will prevent underestimation when we have little reason not to, since we've been waiting on a lot of "things we may add" for a very long time.

Like I'm sorry to be so hot in this thread but you guys really aren't giving us anything of substance and then are consistently surprised when we're frustrated about it. There's a fundamental breakdown of communication here, I think.
I've finally found what I was looking for
A place where I can be without remorse
Because I am a stranger who has found
An even stranger war

Zaolat wrote:WHO THE F*** IS SYBERIS

User avatar
Trotterdam
Postmaster-General
 
Posts: 10541
Founded: Jan 12, 2012
Left-Leaning College State

Postby Trotterdam » Sun Jun 19, 2022 10:19 pm

Racoda wrote:A move to JSON. XML is passé and cumbersome, and in some cases the API returns malformed XML (!).
Both JSON and XML include a bunch of complicated formatting metadata that you ideally need a dedicated parser to make sense of, rather than just dumping the data in an as simple and easy-to-parse format as possible. If I'm going to need to deal with that anyway, I'd rather stick with the format I've already implemented, rather than having to scramble to rewrite every NationStates script ever made to adapt to the suddenly-changed format. I suppose it would be possible to include a toggle parameter to allow individual API users to select whether they want to use the JSON or XML interface, but that would require twice as much code that needs to be kept up-to-date and debugged.

I have noticed the XML interface's poor handling of non-ASCII characters and already reported it long ago, along with two subsequent reminders, but it never got any sort of response, and I can confirm the problem is still there. (Note that it's not exactly malformed XML - it's legal XML that a parser will accept, but just displays a result slightly different than what's intended. There's no rule saying that you can't encode a JSON string in XML, or vice versa, if you really want, although there's no good reason to 99.999% of the time. Is your example different?)

User avatar
United Calanworie
Technical Moderator
 
Posts: 3755
Founded: Dec 12, 2018
Democratic Socialists

Postby United Calanworie » Mon Jun 20, 2022 12:39 am

Trotterdam wrote:
Racoda wrote:A move to JSON. XML is passé and cumbersome, and in some cases the API returns malformed XML (!).
If I'm going to need to deal with that anyway, I'd rather stick with the format I've already implemented, rather than having to scramble to rewrite every NationStates script ever made to adapt to the suddenly-changed format. I suppose it would be possible to include a toggle parameter to allow individual API users to select whether they want to use the JSON or XML interface, but that would require twice as much code that needs to be kept up-to-date and debugged.

I mean realistically that option already exists, see https://www.nationstates.net/pages/api.html#versions for instructions on how to version your queries to only hit one version of the API.
Trans rights are human rights.
||||||||||||||||||||
Discord: Aav#7546 @queerlyfe
She/Her/Hers
My telegrams are not for Moderation enquiries, those belong in a GHR. Feel free to reach out if you want to just chat.

User avatar
Sedgistan
Site Director
 
Posts: 35471
Founded: Oct 20, 2006
Anarchy

Postby Sedgistan » Mon Jun 20, 2022 2:28 am

Syberis wrote:
Eluvatar wrote:I think you might be underestimating how much we may add API-side.


Then perhaps someone should tell us what you may add API-side. After all, it's been a year now since creation of Development Managers, which we were led to believe the whole point of that was to form a pipeline in addition to separation of tasks.

The thread topic is now effectively "what do people want from the API to replicate what they could do with html site scripts" - so the question is really for players to indicate what they want added. I read Elu's post as a suggestion to be more ambitious with requests.

On DMs, none of the roles really cover the API - that's more of an admin area of responsibility, although we'll give feedback on potential changes that impact on the areas we cover.

User avatar
Trotterdam
Postmaster-General
 
Posts: 10541
Founded: Jan 12, 2012
Left-Leaning College State

Postby Trotterdam » Mon Jun 20, 2022 2:59 am

United Calanworie wrote:I mean realistically that option already exists, see https://www.nationstates.net/pages/api.html#versions for instructions on how to version your queries to only hit one version of the API.
Older versions do eventually get retired. And it would be a problem if new shards get added afterwards and those new shards can only be accessed using the new interface instead of using the same format as the old one.

User avatar
Sweeze
Spokesperson
 
Posts: 189
Founded: Oct 21, 2018
Scandinavian Liberal Paradise

Postby Sweeze » Mon Jun 20, 2022 8:21 am

Sedgistan wrote:The thread topic is now effectively "what do people want from the API to replicate what they could do with html site scripts" - so the question is really for players to indicate what they want added. I read Elu's post as a suggestion to be more ambitious with requests.


if this is the case, here's what i would need in the API for shine (swarm's successor) to have feature parity with the current version that is forced to use the html site:
for prepping puppets for an update:
applying to the WA
moving to a specified region

for tagging/detagging regions after update:
changing WFE's of a region
closing embassies of a region
sending embassy from one region to another
uploading and changing regional flag+banners
mass accepting embassies on one region

for making a uniform puppet set:
changing all customizable nation fields (pretitle, demonym, national capital, religion...)
setting custom nation flag
uploading a national banner and disabling all other national banners

misc:
reviving nations
wouldn't be required for feature parity but holy *fuck* json or toml or some other sane format thats not xml would make life so much more convenient
Last edited by Sweeze on Mon Jun 20, 2022 12:44 pm, edited 3 times in total.
| lily supreme command | the mt army third in command | dev of nsdotpy |
[6:38 PM] Chingis: ... the Tom Brady of R/D
5417+ times tag/detag delegate, 5945+ regions hit, first person to become delegate of 200+ regions in an update (and only to do so multiple times)
call me audrey, it/she

User avatar
9003
Diplomat
 
Posts: 624
Founded: Oct 25, 2012
Corporate Police State

Postby 9003 » Mon Jun 20, 2022 8:44 am

To avoid any competitive edge from moving regions you could add a cool down or more effective but more server labor Intense a delay on moving meaning that you send an api command and than 3 seconds later it moves you. This would still allow for scripts to prep puppets quickly and easily with out giving an advantage during update as variance will act up in a 3 second window
proud member of PETZ people for the Ethical Treatment of Zombies

Active member of The cards market place discord

User avatar
[violet]
Executive Director
 
Posts: 16205
Founded: Antiquity

Postby [violet] » Mon Jun 20, 2022 11:55 pm

Racoda wrote:There is a distinction to be made between what I will call passive, reactive and active scripts:

Thanks for those pics -- those tools are really fantastic.

I agree everything you have under "Passive" is outside the scope of this discussion -- they are excellent QoL tools but no requests are being sent to the HTML site. So they're already fine. And "active" scripts should migrate to the API.

"Reactive" contains a bit of a gamut. In the example you link, there are basic key remaps such as "r" for reloading the page, just like you pressed F5. That's a simple navigation shortcut that would make no sense to send to the API. The same goes for things like "Press 'i' to go to your Issues page." However, other elements of that tool are more involved, including some keybinds that issue commands (press "g" to gift a card to your puppet) and a custom puppet builder that's over 200 lines long and generates its own form.

I think the navigation shortcuts are fine, as are keybinds that click things that already exist on the page. But once the tool is crafting its own, new requests (including generating its own forms), or building command-issuing UI elements based on what it read from the page -- i.e. not just acting blindly but having a hand in both sides of the request, both reading and sending -- then those requests should go via the API.

It's hard to delineate this area cleanly, but that seems reasonable to me.

Racoda wrote:This script in particular is impossible to port to the API: there's no "junk card" private command (and I doubt one will ever be added).

Why shouldn't one be added -- because you don't think we should permit this action to be fully automated?

Racoda wrote:A move to JSON. XML is passé and cumbersome, and in some cases the API returns malformed XML (!).

Yes, I agree. I won't promise it now, because I want to keep an achievable minimum viable product, but I do think it's unreasonable not to serve JSON in 2022. Our XML parser is also non-standard, which is why it produces bogus output sometimes. Although the specific bug you mention is probably related to our character encoding rather than the parser.

Racoda wrote:A possibility to authenticate differently than through headers (ex. a POST request with a password field). For WebApps, a CORS request is sent first if the app tries to send a request containing non-standard headers (like X-Password). However, the app is not notified if/when a CORS request is sent and those count towards the rate limit.

I've been reluctant to mess with CORS because I'm not very familiar with it and it's full of security issues. But yes, I think this is very possible, as is addressing the issue you link. You would have to provide me with specific instructions on what you want, like you're talking to a five year old who isn't very familiar with CORS.

User avatar
Trotterdam
Postmaster-General
 
Posts: 10541
Founded: Jan 12, 2012
Left-Leaning College State

Postby Trotterdam » Tue Jun 21, 2022 12:40 am

[violet] wrote:Although the specific bug you mention is probably related to our character encoding rather than the parser.
It's not a character encoding issue as such - both "\u00AE" (C/JSON/etc.) and "®" (HTML/XML/etc.) are, in their relevant contexts, correct character references that explicitly reference Unicode, as opposed to some other character encoding such as ISO 8859-1 (which would just be "\xAE" in C/JSON/etc., and cannot be directly specified in XML at all except when the entire page is in ISO 8859-1 encoding and you're just serving the raw binary code). It's just that at some point, some part of the code appears to go "hey, this is a Unicode character and I should ASCIIfy it" before the XML constructor appears to ever see it, so it doesn't know that what it's writing will eventually be used for XML and it picks the wrong type of ASCIIfication. I'm guessing whatever piece of code does this is also likely to mangle backslashes as "\" -> "\\", but I can't test this because backslashes are forbidden in custom fields anyway, and they don't seem to appear anywhere in canon.

User avatar
SherpDaWerp
Technical Moderator
 
Posts: 1895
Founded: Mar 02, 2016
Benevolent Dictatorship

Postby SherpDaWerp » Tue Jun 21, 2022 2:24 am

[violet] wrote:I've been reluctant to mess with CORS because I'm not very familiar with it and it's full of security issues. But yes, I think this is very possible, as is addressing the issue you link. You would have to provide me with specific instructions on what you want, like you're talking to a five year old who isn't very familiar with CORS.

On the CORS topic; while I agree with Racoda's request, what would really help is to exempt OPTIONS requests from the ratelimit, as mentioned here by Elu. That just (well, "just" as much as anything on this website) involves detecting the request type before any processing happens and sending only the specific, potentially hard-coded CORS headers (Access-Control-Allow-Origin, Access-Control-Allow-Methods, Access-Control-Allow-Headers, Access-Control-Max-Age) rather than a full API response (and note that user-agent isn't a special header anymore so you should make sure its on the allowed headers list too).

Racoda's suggestion to optionally add authentication to the body of a POST request would be nice, as it could give us a work-around for the CORS issue and would give a much more web-standard system for authentication, but exempting OPTIONS altogether would be better purely in terms of stopping the CORS problem.


In terms of a laundry list of what would need to be moved off the HTML site in order to not impact Cards scripts, you're mostly just looking at
  • collection management (add/remove cards)
  • puppet creation/management (i.e. changing fields, nation flags, banners, puppet creation, moving)
  • ...and the big one, issues, which no-one wants to see become automated, so this one is less a "new api addition" and more a "new way to make cards issues-less", which is a whole thread worth of discussion in and of itself
Most other Cards scripts that I'm aware of are more on the line of passive/reactive in terms of either adding info to card pages from the API or from player-held servers, or keybinding and queueing cards to process.
Last edited by SherpDaWerp on Tue Jun 21, 2022 2:31 am, edited 1 time in total.
Became an editor on 18/01/23 techie on 29/01/24

Rampant statistical speculation from before then is entirely unofficial

User avatar
Haganham
Minister
 
Posts: 3065
Founded: Aug 17, 2021
Psychotic Dictatorship

Postby Haganham » Tue Jun 21, 2022 5:09 am

Just kidding. Web3 isn't real.

How can max say this when we were minting NFTs in 2018 :roll:
Imagine reading a signature, but over the course of it the quality seems to deteriorate and it gets wose an wose, where the swenetence stwucture and gwammer rewerts to a pwoint of uttew non swence, an u jus dont wanna wead it anymwore (o´ω`o) awd twa wol owdewl iws jus awfwul (´・ω・`);. bwt tw sinawtur iwswnwt obwer nyet, it gwos own an own an own an own. uwu wanyaa stwop weadwing bwut uwu cwant stop wewding, uwu stwartd thwis awnd ur gwoing two fwinibsh it nowo mwattew wat! uwu hab mwoxie kwiddowo, bwut uwu wibl gwib ub sowon. i cwan wite wike dis fwor owors, swo dwont cwalengbe mii..

… wbats dis??? uwu awe stwill weedinb mwie sinatwr?? uwu habe awot ob detewemwinyanyatiom!! 。◕‿◕。! u habve comopweedid tha signwtr, good job!

User avatar
Roavin
Admin
 
Posts: 1777
Founded: Apr 07, 2016
Democratic Socialists

Postby Roavin » Tue Jun 21, 2022 12:31 pm

To add onto the JSON argument: That transition can also be used as an opportunity to represent data more idiomatically, i.e. having something like
Code: Select all
"endorsements": [
   "testlandia",
   "maxtopia"
],

rather than
Code: Select all
<ENDORSEMENTS>testlandia:maxtopia</ENDORSEMENTS>

... which requires additional parsing. Not difficult parsing, but additional parsing nonetheless — it's better to have the API endpoint do the nominal transformation.
Helpful Resources: One Stop Rules Shop | API documentation | NS Coders Discord
About me: Longest serving Prime Minister in TSP | Former First Warden of TGW | aka Curious Observations

Feel free to TG me, but not about moderation matters.

User avatar
Racoda
Technical Moderator
 
Posts: 579
Founded: Aug 12, 2014
Democratic Socialists

Postby Racoda » Wed Jun 22, 2022 7:57 pm

[violet] wrote:"Reactive" contains a bit of a gamut. In the example you link, there are basic key remaps such as "r" for reloading the page, just like you pressed F5. That's a simple navigation shortcut that would make no sense to send to the API. The same goes for things like "Press 'i' to go to your Issues page." However, other elements of that tool are more involved, including some keybinds that issue commands (press "g" to gift a card to your puppet) and a custom puppet builder that's over 200 lines long and generates its own form.

Yes, that's an oversight on my part -- I forgot that script includes a puppet creator. That is obviously in the "active" and not "reactive" category.




[violet] wrote:
Racoda wrote:This script in particular is impossible to port to the API: there's no "junk card" private command (and I doubt one will ever be added).

Why shouldn't one be added -- because you don't think we should permit this action to be fully automated?

SherpDaWerp wrote:
  • ...and the big one, issues, which no-one wants to see become automated, so this one is less a "new api addition" and more a "new way to make cards issues-less", which is a whole thread worth of discussion in and of itself

Mainly this, having fully automated cards farming (fully-automated issue answering as well as fully-automated junking) is a bad idea. Automated junking through HTML scripts is currently forbidden, and if there's going to be an API to junk cards, I think similar restrictions should be imposed (maybe a mandatory userclick/useraction parameter that's checked server-side?).




[violet] wrote:I've been reluctant to mess with CORS because I'm not very familiar with it and it's full of security issues. But yes, I think this is very possible, as is addressing the issue you link. You would have to provide me with specific instructions on what you want, like you're talking to a five year old who isn't very familiar with CORS.

To be honest, I have no experience with CORS -- hence the suggestion to authenticate differently than in headers. I've had a discussion with Sherp after posting and I agree that exempting OPTIONS from the rate-limit (and preventing it from sending an API response) is the best solution, especially since a CORS request will be needed anyway to authorize sending a custom user-agent header. Currently the API returns:
Code: Select all
access-control-allow-headers: X-Pin, X-Password, X-Autologin

which should also include user-agent to enable WebApps to set a user-agent. Note however that there is a nearly 7-year old chromium bug that prevents setting the user-agent on that browser. The one NS JavaScript library I know of goes around this by identifying itself in the URL -- that might remain the only option as long as chromium is bugged.

Acting as a player unless accompagnied by mod action or reddish text
Any pronouns

User avatar
Flanderlion
Minister
 
Posts: 2226
Founded: Nov 25, 2013
Psychotic Dictatorship

Postby Flanderlion » Thu Jun 23, 2022 2:59 am

If we're saying what we want from API, sending API TGs using stamps, nations a nation is endorsing list (ridiculous that a nation would not know IC who it says is good), ability to create TG templates by API, and larger dumps of censuses (can just be refreshed when the daily census chooses it) rather than having to do Valentine esque style of just a gazillion requests. Also ability to request in JSON (but keep XML as default just so old scripts don't have to do anything). Also view contents of TGs.

For others, moving/endorsing/etc. all automatic could be done, just not sure if it's really desirable to automate everything. Could require a manual login within 24 hours (or 7 days) for the endo to count?
As always, I'm representing myself.
Information
Wishlist

User avatar
Refuge Isle
Technical Moderator
 
Posts: 1873
Founded: Dec 14, 2018
Left-wing Utopia

Postby Refuge Isle » Thu Jun 23, 2022 10:36 am

Flanderlion wrote:For others, moving/endorsing/etc. all automatic could be done, just not sure if it's really desirable to automate everything.

Nobody in r/d is keen to see our game reduced to automated scripts. There is no point of it without the human element -- skill, reaction time, dedication.

It would also be kinda wonk if your region got invaded and destroyed by a bot operating independently.

PreviousNext

Advertisement

Remove ads

Return to Technical

Who is online

Users browsing this forum: Argeentina, Armisael, Bla-Haj, Dushina, Elwher, Heromerland, IdontCare, Idontknowium, Lands of Ann, Lunayria, North American Imperial State, The Blazing Aura, The Isles of Inbetween, Umbratellus, Usual People In Life, Wattsland, Xoshen

Advertisement

Remove ads