[Tech Ruling Request] Command-Line Browsers

[United Calanworie]

As per this thread, Selenium and the like are not considered browsers.

However, is a tool like Lynx which is designed for human use considered a browser? I'm considering doing a few projects that would entail the use of Lynx as a browser, but wanted to find out, in light of the ruling on Selenium, as to whether or not it would be considered a browser.

Thanks for your time!

[United Calanworie]

4 day bump.

[[violet]]

I'm finding this angle of questioning a bit strange. The Script Rules detail what is permissible when interacting with the HTML site. It's very restrictive, because wherever possible, all automated or script-like behavior should be happening on the API, not the HTML site. All tools are treated the same way, whether they're scripts, bots, browser add-ons, whatever. So I'm not sure why we need to hair-split the definition of "browser."

[CoraSpia]

I'm finding this angle of questioning a bit strange. The Script Rules detail what is permissible when interacting with the HTML site. It's very restrictive, because wherever possible, all automated or script-like behavior should be happening on the API, not the HTML site. All tools are treated the same way, whether they're scripts, bots, browser add-ons, whatever. So I'm not sure why we need to hair-split the definition of "browser."

Lynx isn't a browser add-on, it's a very early text-based browser. I'm not sure who still uses it (aside from the OP,) but it definitely doesn't give you an unfair advantage...in fact it's extremely slow and clunky.

[[violet]]

Yeah, I use it myself sometimes, like for quickly testing whether the site is up. I just don't want to confuse the issue because I don't think it matters whether we call it a browser or not.

[United Calanworie]

I'm finding this angle of questioning a bit strange. The Script Rules detail what is permissible when interacting with the HTML site. It's very restrictive, because wherever possible, all automated or script-like behavior should be happening on the API, not the HTML site. All tools are treated the same way, whether they're scripts, bots, browser add-ons, whatever. So I'm not sure why we need to hair-split the definition of "browser."

I wasn't sure whether or not Lynx fell into the same boat as Selenium, because of the fact that it's a command-line tool. That's all. Sorry for any confusion.

[Flanderlion]

I'm finding this angle of questioning a bit strange. The Script Rules detail what is permissible when interacting with the HTML site. It's very restrictive, because wherever possible, all automated or script-like behavior should be happening on the API, not the HTML site. All tools are treated the same way, whether they're scripts, bots, browser add-ons, whatever. So I'm not sure why we need to hair-split the definition of "browser."

The angle of questioning from various directions is primarily for N day and how to push production as fast as possible without breaking the rules. This includes stuff like containers, upgraded NS++ type scripts and other stuff.

[SherpDaWerp]

I'm finding this angle of questioning a bit strange. The Script Rules detail what is permissible when interacting with the HTML site. It's very restrictive, because wherever possible, all automated or script-like behavior should be happening on the API, not the HTML site. All tools are treated the same way, whether they're scripts, bots, browser add-ons, whatever. So I'm not sure why we need to hair-split the definition of "browser."

The angle of questioning from various directions is primarily for N day and how to push production as fast as possible without breaking the rules. This includes stuff like containers, upgraded NS++ type scripts and other stuff.

Additionally, two new Issues scripts tools (for Cards gameplay) are in development.

We're just trying to make sure we're not breaking any rules, and while that might come across like "trying to find loopholes", we're blurring the line between automated and manual (my Selenium ruling request, for instance - despite being "manual" with a person pressing a button for each action, due to its scripting-focused design it was found to be illegal), so it's preferable to get clarification rather than break rules and find out later when you get warned/deat.

[[violet]]

Yep, for sure. I'm still a bit confused as to why you want a definition of a "browser," since this isn't mentioned in the Script Rules. A script isn't a browser but is bound by the same rules as browser add-ons.

Additionally, two new Issues scripts (for Cards gameplay) are in development.

With few exceptions, it's not permissible to use a script to answer issues outside of the API at all. Also, of course, scripts may not automatically do anything with Cards.

[SherpDaWerp]

Additionally, two new Issues scripts (for Cards gameplay) are in development.

With few exceptions, it's not permissible to use a script to answer issues outside of the API at all. Also, of course, scripts may not automatically do anything with Cards.

gah, the specter of word choice forever haunts me...

The "script" portion just generates links, the actual answering is done by a human using a browser to open those links (and a userscript ensures that the user only opens one link at a time).

[United Calanworie]

Yep, for sure. I'm still a bit confused as to why you want a definition of a "browser," since this isn't mentioned in the Script Rules. A script isn't a browser but is bound by the same rules as browser add-ons.

Because from what I understand, vis a vis the script rules, if a tool (let's take Breeze as an example) merely adds keybinds for an action, or moves a button on the page, it just needs to abide by 1:1, not 10 requests per minute. I was thinking about using Lynx to reduce page load times (as it won't load any fancy CSS) and creating a tool that (on user input) navigates directly to a specific page. It doesn't interact with anything, it just loads a page, like how Breeze can bring up the main defender jumppoint with a single press of the "B" key. However, I was wary of exactly whether or not it would be considered a script vs an add-on, based on the Selenium ruling, and that you couldn't generate heaps of links for users to cycle through, and action, with a single keypress.

tl;dr I was wondering if I wrote a browser add-on for Lynx, whether it's an add-on, or a script (due to the browser's nature). Sorry, I should have been more clear in my OP.

[Trotterdam]

What I want to know is how Lynx is considered a "command-line" tool. It's run in a text console, sure, but it's a persistent program with a two-dimensional user interface (even if it's a text-only one), not something like wget which is actually controlled from the system command line (but which I wouldn't call a browser), or even something like traditional FTP clients which implement their own command line.

[[violet]]

Yep, for sure. I'm still a bit confused as to why you want a definition of a "browser," since this isn't mentioned in the Script Rules. A script isn't a browser but is bound by the same rules as browser add-ons.

Because from what I understand, vis a vis the script rules, if a tool (let's take Breeze as an example) merely adds keybinds for an action, or moves a button on the page, it just needs to abide by 1:1, not 10 requests per minute. I was thinking about using Lynx to reduce page load times (as it won't load any fancy CSS) and creating a tool that (on user input) navigates directly to a specific page. It doesn't interact with anything, it just loads a page, like how Breeze can bring up the main defender jumppoint with a single press of the "B" key. However, I was wary of exactly whether or not it would be considered a script vs an add-on, based on the Selenium ruling, and that you couldn't generate heaps of links for users to cycle through, and action, with a single keypress.

tl;dr I was wondering if I wrote a browser add-on for Lynx, whether it's an add-on, or a script (due to the browser's nature). Sorry, I should have been more clear in my OP.

Aha, I think I get it. So let me see if I have this right: The concern is that if the tool is built on Lynx, then every single request sent by Lynx during regular browsing might be considered to be from the tool itself, and therefore need to remain within the 10 requests per minute limit. And this is particularly true for requests that come about by the user clicking on a link that was created by the tool. Whereas if the tool is considered to be an add-on to the Lynx browser, then only special additional requests it actually transmits itself (if any) need to be counted within that ratelimit. Is that correct?

e: "Clicking" is maybe not the greatest choice of word here, but you know what I mean.

[United Calanworie]

Because from what I understand, vis a vis the script rules, if a tool (let's take Breeze as an example) merely adds keybinds for an action, or moves a button on the page, it just needs to abide by 1:1, not 10 requests per minute. I was thinking about using Lynx to reduce page load times (as it won't load any fancy CSS) and creating a tool that (on user input) navigates directly to a specific page. It doesn't interact with anything, it just loads a page, like how Breeze can bring up the main defender jumppoint with a single press of the "B" key. However, I was wary of exactly whether or not it would be considered a script vs an add-on, based on the Selenium ruling, and that you couldn't generate heaps of links for users to cycle through, and action, with a single keypress.

tl;dr I was wondering if I wrote a browser add-on for Lynx, whether it's an add-on, or a script (due to the browser's nature). Sorry, I should have been more clear in my OP.

Aha, I think I get it. So let me see if I have this right: The concern is that if the tool is built on Lynx, then every single request sent by Lynx during regular browsing might be considered to be from the tool itself, and therefore need to remain within the 10 requests per minute limit. And this is particularly true for requests that come about by the user clicking on a link that was created by the tool. Whereas if the tool is considered to be an add-on to the Lynx browser, then only special additional requests it actually transmits itself (if any) need to be counted within that ratelimit. Is that correct?

e: "Clicking" is maybe not the greatest choice of word here, but you know what I mean.

Yes, that is absolutely my concern.

[Roavin]

I think some unintentional confusion has arisen because of the questions that were asked, both in this thread and the previous thread about Selenium.

SherpDaWerp had asked if Selenium was considered a browser. [v] answered no, but didn't otherwise specify whether Sherp's plan was problematic as per site rules or not. Aav's question seems to be based on Sherp's interpretation of [v]'s ruling, but I suspect strongly that [v]'s ruling was just answering Sherp's direct question (is Selenium a browser) and not whether Sherp's plan (which is to do a bunch of things quicker than the rate limit so long as each single thing is in response to direct user input). My suspicion is supported by [v]'s answer in this thread, wondering by the question of browsers is now coming up.

[v], can you clarify that your ruling on Selenium was meant to narrowly answer Sherp's question, and not actually a judgement call on what Sherp intended to do?

On the subject matter itself: It had been my understanding that a single action that happens immediately in response to user input is not bound by rate so long as it either uses an existing unmodified button in a browser or follows simultaneity, and is not an action that is not explicitly restricted from non-API stuff, such as anything TG-related. Under that understanding, both OP's plan and Sherp's plan are fine so long as they take care to follow the simultaneity rule.

On the other hand, if my understanding is wrong, it would throw into question the legality of quite a few things that have been developed in the past few years.

[[violet]]

[v], can you clarify that your ruling on Selenium was meant to narrowly answer Sherp's question, and not actually a judgement call on what Sherp intended to do?

This is correct! I don't read code and sign off on individual scripts. I only answer questions about the rules.

On the subject matter itself: It had been my understanding that a single action that happens immediately in response to user input is not bound by rate so long as it either uses an existing unmodified button in a browser or follows simultaneity, and is not an action that is not explicitly restricted from non-API stuff, such as anything TG-related. Under that understanding, both OP's plan and Sherp's plan are fine so long as they take care to follow the simultaneity rule.

I think that's true under the rules today.

But I'm getting the feeling that we have a problem with tools that generate a list of clickable links and present them to a human in a format suitable for fastest possible clicking/button-mashing. In some form, these have been around for a long time, especially in R/D and region recruiting. But it sounds to me like there are more (Trading Cards-related) scripts in development that are essentially automated systems employing a human for some mindless button-pressing monkey work. This is pretty far from the original intent of the rules, which is to permit users to add theming to the site, or accessibility, or minor functionality - everything else being directed to the API. And these tools are hitting the HTML site rather than the API not because they're enhancing an experience for humans, but because they want to dodge script-specific restrictions - in the case of Trading Cards, the fact that bots don't generate them when answering issues.

Scripts hitting the HTML site are a pain in the butt: They make themselves look like regular human traffic, but have a much bigger impact on site performance. It's very hard to restrict them without also making things worse for our regular users.

I guess the intent of the headless/command-line browser system is to streamline this process even further, so humans don't even need to open a real browser after their links are generated. This would create a bad outcome for all concerned: the user isn't really playing the game or enjoying anything, the API isn't being used, and the site isn't getting any genuine traffic or ad revenue, but is tied up serving expensive HTML requests.

So I don't want to end up there. I'm not sure what the best way is to avoid it, though; whether we have to ban scripts that generate lists of HTML-site links or make them responsible for rate-limiting them or what.

[Trotterdam]

The problem is defining at what point a script's features change from "added functionality that is still consciously controlled by the user" to "automated links that the user only technically operates via mindless clicking", since I think it's more of a sliding scale rather than a hard distinction. If you can define the difference then it's easy enough to to say the latter is banned, though harder to enforce the ban.

[SherpDaWerp]

It's very hard to restrict them without also making things worse for our regular users.

I'd hope responsible script developers would at least consider a "manual" approach - write their scripts in a very deliberately non-rule-breaking way rather than force the creation of some complicated activity tracker or similar to sniff out rulebreakers.

I guess the intent of the headless/command-line browser system is to streamline this process even further, so humans don't even need to open a real browser after their links are generated. This would create a bad outcome for all concerned: the user isn't really playing the game or enjoying anything, the API isn't being used, and the site isn't getting any genuine traffic or ad revenue, but is tied up serving expensive HTML requests.

This is pretty much exactly what I was planning, so it's good that I checked. For me, at least, "enjoyment" left large-scale farming a little while ago, and I doubt many others find it interesting to farm - the results make the incentive, not the process. That said, while I was aware of that "enjoyment" aspect when I made my ruling request, I hadn't considered the loss of ad revenue to the site from using a headless browser.

I'm not sure what the best way is to avoid it, though; whether we have to ban scripts that generate lists of HTML-site links or make them responsible for rate-limiting them or what.

I'm open to accepting a ratelimit on my link-generator script (it would be a relatively easy extension to my simultaneity enforcement system), ease of process matters far more to me in farming than time spent (though I'd imagine for those out there with hundreds to thousands of puppets, time spent is also a concern). I'd certainly prefer ratelimiting over wholesale prohibition.

I hope it would be a less restrictive ratelimit than the pure HTML scripting limit of 10/minute, given that there is still technically user input involved (and, at least for mine, the user has the option to specify issue options to answer beforehand, not just randomly choose).

Finally, I think it's worth noting that the two cards-based link-generator scripts actually significantly reduce the number of HTML pages loaded for each nation, AFAIK. The traditional setup is <nation login> (<issues> <issue> <answer>)x5, whereas link-generator scripts follow <nation login> (<answer>)x5, loading 10 less pages per nation (and containers, by virtue of saving cookies for each puppet, skip the <nation login> step as well). From my perspective, it seems like the best outcome for site loads (at least in terms of # of pages loaded) would involve link-generator scripts of some description.

[Trotterdam]

For me, at least, "enjoyment" left large-scale farming a little while ago, and I doubt many others find it interesting to farm - the results make the incentive, not the process.

I don't, but I abandoned the card minigame with extreme prejudice as soon as it stopped being an April Fools joke. I recognized it immediately as the sort of thing designed to be addictive without actually being fun (exactly what the original April Fools event was parodying!) and I want no part in that.

The traditional setup is <nation login> (<issues> <issue> <answer>)x5, whereas link-generator scripts follow <nation login> (<answer>)x5, loading 10 less pages per nation (and containers, by virtue of saving cookies for each puppet, skip the <nation login> step as well).

How do you answer an issue without loading the issue? You don't know what options it has. (Even option 1 isn't guaranteed to always be available.)

[SherpDaWerp]

For me, at least, "enjoyment" left large-scale farming a little while ago, and I doubt many others find it interesting to farm - the results make the incentive, not the process.

I don't, but I abandoned the card minigame with extreme prejudice as soon as it stopped being an April Fools joke. I recognized it immediately as the sort of thing designed to be addictive without actually being fun (exactly what the original April Fools event was parodying!) and I want no part in that.

I'm driven more by want of completion than anything else at this point, and I don't think I'll go for a similarly expensive collection in future seasons exactly for this reason.

The traditional setup is <nation login> (<issues> <issue> <answer>)x5, whereas link-generator scripts follow <nation login> (<answer>)x5, loading 10 less pages per nation (and containers, by virtue of saving cookies for each puppet, skip the <nation login> step as well).

How do you answer an issue without loading the issue? You don't know what options it has. (Even option 1 isn't guaranteed to always be available.)

There is an initial script that uses the API to get issue numbers and option numbers for each puppet, then it outputs a html link list for the user to click. If you want to have a look, the code is here.

[Trotterdam]

There is an initial script that uses the API to get issue numbers and option numbers for each puppet, then it outputs a html link list for the user to click.

Oh right. Reading issues is API-enabled now! Clever.

(And answering issues is API-enabled too, but it disables cards.)

Also it occurs to me that, regarding ratelimits, since the rule is officially listed as "10 per minute" rather than "once per 6 seconds", you could answer issues for a small number of nations instantly even with ratelimits. It'll be six hours before you get more issues.

[[violet]]

It's very hard to restrict them without also making things worse for our regular users.

I'd hope responsible script developers would at least consider a "manual" approach - write their scripts in a very deliberately non-rule-breaking way rather than force the creation of some complicated activity tracker or similar to sniff out rulebreakers.

Most scripts we see are written by people who want to follow the rules. Well, actually, that's not true if you include Facebook, Google et al, because they have bot armies that swarm over the site no matter what we do. But it's pretty rare that I find a bot written for NS that breaks the rules on purpose.

More common are bots that go haywire and the author doesn't seem to notice, making nutty amounts of requests, often for the same thing over and over. That is pretty common.

The user scripts (or link-generating scripts) are different because they don't identify themselves: Their traffic arrives as browser clicks from genuine humans, with no identifying info in their UserAgent string. Mixed in with regular site traffic, of which we get a lot, they're hard to spot, let along manage.

Finally, I think it's worth noting that the two cards-based link-generator scripts actually significantly reduce the number of HTML pages loaded for each nation, AFAIK. The traditional setup is <nation login> (<issues> <issue> <answer>)x5, whereas link-generator scripts follow <nation login> (<answer>)x5, loading 10 less pages per nation (and containers, by virtue of saving cookies for each puppet, skip the <nation login> step as well). From my perspective, it seems like the best outcome for site loads (at least in terms of # of pages loaded) would involve link-generator scripts of some description.

I see where you're coming from, but I'd say it's less about pages than efficiency. We have no problem if players load a whole bunch of pages while having fun on the site. That's why we're here. But the situation we seem to have found ourselves in is with that each card-harvesting click, the site generates a page full of stats and rankings--which trigger re-rankings of up to 200,000 other nations across 85 census scales--none of which you care about. You want to make that process as fast as possible, naturally enough, but the more you succeed, the more people will do it. And, of course, it's happening in the HTML channel, which is less efficient than the API. (Or both! If you are authenticating the nation in the API to check its issues and then re-authenticating in the HTML site to answer them.) So it's not just a lot of server load, it's a lot of load for nothing.

It's good if Trading Cards can lure people into answering more issues and paying attention to their storylines. It's not great if people are using autogenerator tools to spam requests for pages that nobody reads--not great for us and not much fun for you. Of course, as you say, the fun is in finding cards, so I understand the motivation. We just need a better way of managing that. Like maybe you don't automatically gather a pack when one drops after answering an issue, but have to actually click something to claim it.

[9003]

actually click something to claim it.

This would only add one line of code and an extra few millasecounds to most users of the code (a minority of players causing a lot of requests)

A better way is making card packs only spawnable 1 pack per secound. If there is no reason to go faster then 1 issue a secound it fixes the issue. The 1 secound being a flexible time that hopfuly is between the average read time of an issue for regular players and a comfy speed for the server to handle the requests.

While hard to inforce the system sudo does it by not incentivizing actions faster then x speed. (part of this would be making said announcement known so people push to that line rather then push to the 0 secound issue answer)

[Ever-Wandering Souls]

But I'm getting the feeling that we have a problem with tools that generate a list of clickable links and present them to a human in a format suitable for fastest possible clicking/button-mashing. In some form, these have been around for a long time, especially in R/D and region recruiting. But it sounds to me like there are more (Trading Cards-related) scripts in development that are essentially automated systems employing a human for some mindless button-pressing monkey work. This is pretty far from the original intent of the rules, which is to permit users to add theming to the site, or accessibility, or minor functionality - everything else being directed to the API. And these tools are hitting the HTML site rather than the API not because they're enhancing an experience for humans, but because they want to dodge script-specific restrictions - in the case of Trading Cards, the fact that bots don't generate them when answering issues.
--snip--
I guess the intent of the headless/command-line browser system is to streamline this process even further, so humans don't even need to open a real browser after their links are generated. This would create a bad outcome for all concerned: the user isn't really playing the game or enjoying anything, the API isn't being used, and the site isn't getting any genuine traffic or ad revenue, but is tied up serving expensive HTML requests.

So I don't want to end up there. I'm not sure what the best way is to avoid it, though; whether we have to ban scripts that generate lists of HTML-site links or make them responsible for rate-limiting them or what.

Emphasis mine.

As a user, my god, this. I mean, the issues with load and ad revenue are serious too, and I respect that, but in terms of experience, so much this. Ease of use/Quality of Life tools are nice; script arms warfare that renders the game noncompetitive to anyone not willing to use an obscure optimization tool based on a text-only Usenet browser that's a decade or two older than they are... is truly jumping the shark in terms of inaccessibility to newcomers.

[Ballotonia]

Finally, I think it's worth noting that the two cards-based link-generator scripts actually significantly reduce the number of HTML pages loaded for each nation, AFAIK. The traditional setup is <nation login> (<issues> <issue> <answer>)x5, whereas link-generator scripts follow <nation login> (<answer>)x5, loading 10 less pages per nation (and containers, by virtue of saving cookies for each puppet, skip the <nation login> step as well). From my perspective, it seems like the best outcome for site loads (at least in terms of # of pages loaded) would involve link-generator scripts of some description.

The 'for each nation' is a flawed metric. What I see happening is that as the scripting becomes more efficient in terms of time spent the farmers expand their farms with extra nations. Any fewer calls per nation made are thus not spent as less load on the server but end up as extra packs generated by additional farm puppets with the total number of calls to the server being the same.

Ballotonia