NATION

PASSWORD

Roavin's big list of advanced API/script questions

Bug reports, general help, ideas for improvements, and questions about how things are meant to work.
User avatar
Roavin
Admin
 
Posts: 1777
Founded: Apr 07, 2016
Democratic Socialists

Roavin's big list of advanced API/script questions

Postby Roavin » Fri May 19, 2017 3:19 pm

Hi,




1. Are two simultaneous non-API non-restricted queries based on a single user action permissible?
To give a trivial example, suppose I wrote a browser extension that compared two nations side-by-side. Now let's suppose the user wants to change to another tab (from "People" to "Government", for example) - in this case, both halves would require a request, though intuitively it would be based on a single user input.
To give another example: Suppose I'm building a special happenings page with one filter for the world, but a wider filter for a specific region or nation; as an example, maybe I want to see worldwide moves but only see endorsements coming from The North Pacific. This could be implemented by making two queries to the ajax-feed and interleaving the result data based on the happenings ID. If that page included a manual "refresh" button, would it be permissible that this button then queries the ajax-feed twice based on that single click?




2. Is API-usage "per-organization" allowed as opposed to "per-user", and how should this best be reflected in the User-Agent?
Context: My cloud server hosts 5 different Discord bots spanning 16 different Discord servers, with 5 regions represented and 3 of those servers being NS-interregional (such as the NSGP Discord Server). All four of these bots include an extension for pulling up information on a nation or a region using an API-call (either using Darcania's cogs or one I rolled on my own a while back before Darcania wrote his). Now obviously, these together are rate-limited as a unit as they all come from the same server, and that's okay (and they all identify as "Roavin" in the User-Agent at this time). But my worry is something like this - suppose in one of the Discord servers, several individuals begin showing off their nations to each other (not an infrequent occurrence), leading to something like 10-15 requests being made in a minute. At the same time, I'm using the API at home in some capacity, such as querying the last 12 hours of happenings or running the tool 20XX which makes a request once a second. This combination won't hit the rate limit imposed by the NationStates server, as their origin is different, but they both identify as Roavin and would in sum exceed the limit for me personally. The API documentation says "Do Not Sneakily Exceed Rate Limits"; while the intent surely isn't to sneakily exceed the rate limits, given the explanation in the API documentation, technically this could lead to a situation where I de facto exceed the rate limit (albeit unintentionally).




3. The simultaneity rule applies to restricted actions. Do any similar restrictions exist for non-restricted actions (beyond "be reasonable with our server")?
I don't intend to start flooding NationStates with a bunch of simultaneous requests - rather, I'd handle non-restricted actions as if the simultaneity rule applied anyway. This is just out of personal curiosity.




4. Is logging in a restricted action?
While it doesn't generate a world happening, it does update the "Last Government Activity" rating, so intuitively I would think yes. On the other hand, login scripts are permissible, so the precedent seems to be no.




5. Is it permissible to satisfy the script simultaneity rule by manually spacing out user actions?
Now, by a strict legalistic reading of viewtopic.php?t=371830&p=28094530 it's not permissible, but to explain what I mean: Suppose I make a page with several buttons that perform restricted actions that the user wishes to perform sequentially. To give a realistic example relevant to R/D, suppose there are three buttons for move to a specific region, resign from WA, apply to WA (a standard procedure amongst defenders). Now, as it's written, the simultaneity rule states that when clicking one of the buttons, the other two buttons must be disabled until the current request has fully completed. My question is - what if instead, the other buttons remained clickable, and the page internally places those actions in a queue with a timestamp of that click. Whenever a request completes, the next request from the queue is then issued, with the appropriate userclick-parameter (as described by Eluvatar here) to identify when the actual click was. So the workflow for the user would just be clicking those three appropriately labeled buttons in succession however fast they are physically able; the backend code makes sure only one request sends at a time, in each case with when the actual click was performed given as parameter.




6. Is editing a dispatch a restricted action?
At this time, only the author nation can edit a dispatch. So when a dispatch contains something like operation logs, the burden currently falls upon that one individual to keep these logs up to date, unless the nation is shared (and I know mods don't like that). An idea I had was to keep these records on our offsite forum, where we can set fine-grained permissions on who can edit the operation logs. Those of us who would be editing and adding to these operation logs would be on the off-site forum anyway. It's important to us that our regional nations that are on-site only can see these logs too, so the idea entails writing a script to run on a trigger or timer or such which reads the forum, does the appropriate changes wrt BBCode-incompatibilities, and then POSTs that into a pre-created dispatch.
Creating a dispatch creates a happening, so the target dispatch would have to be manually created (as this counts as a restricted action). Meanwhile, editing a dispatch does not cause a a happening, so a strict legalistic reading of the rules would allow this.




Thanks :)

-- Roavin
Helpful Resources: One Stop Rules Shop | API documentation | NS Coders Discord
About me: Longest serving Prime Minister in TSP | Former First Warden of TGW | aka Curious Observations

Feel free to TG me, but not about moderation matters.

User avatar
[violet]
Executive Director
 
Posts: 16205
Founded: Antiquity

Postby [violet] » Fri May 19, 2017 8:31 pm

Roavin wrote:1. Are two simultaneous non-API non-restricted queries based on a single user action permissible?

Yep. Only restricted actions cannot run simultaneously.

Roavin wrote:2. Is API-usage "per-organization" allowed as opposed to "per-user", and how should this best be reflected in the User-Agent?

It's per user, so different people using the same tool for different purposes all have their own bandwidth allowances. But that's assuming they're acting independently. In the situation where there's a site collecting information from us to display to their visitors, for example, we consider the site to be the user, not their visitors.

Roavin wrote:3. The simultaneity rule applies to restricted actions. Do any similar restrictions exist for non-restricted actions (beyond "be reasonable with our server")?

No, just what's in the Script Rules, primarily the much lower rate limit.

Roavin wrote:4. Is logging in a restricted action?

No, as it doesn't cause anything to change except your own nation.

Roavin wrote:5. Is it permissible to satisfy the script simultaneity rule by manually spacing out user actions?

No, because queuing isn't "immediately responding to a user's click" as required by the Script Rules.

Roavin wrote:6. Is editing a dispatch a restricted action?

No. Neither is publishing a dispatch, as it only generates a Happenings line in your own nation. [EDIT: This is incorrect: see below.]
Last edited by [violet] on Sun May 21, 2017 5:16 pm, edited 1 time in total.

User avatar
Kethania
Lobbyist
 
Posts: 21
Founded: Nov 01, 2015
Left-wing Utopia

Postby Kethania » Sat May 20, 2017 4:05 am

[violet] wrote:
Roavin wrote:6. Is editing a dispatch a restricted action?

No. Neither is publishing a dispatch, as it only generates a Happenings line in your own nation.

Does this also apply if a dispatch is pinned in a region?
It is clear that pinning can only be done manually, as it generates a regional happening, but can a script automatically edit already pinned dispatches?
Last edited by Kethania on Sat May 20, 2017 4:08 am, edited 1 time in total.

User avatar
Leppikania
Minister
 
Posts: 2332
Founded: Apr 13, 2015
Left-wing Utopia

Postby Leppikania » Sat May 20, 2017 7:51 am

[violet] wrote:
Roavin wrote:6. Is editing a dispatch a restricted action?

No. Neither is publishing a dispatch, as it only generates a Happenings line in your own nation.

I asked this question previously, and Ballotonia said it was a restricted action. Could you perhaps discuss that?

And also, publishing a dispatch could generate a happenings line on somone else's notices page.

Kethania wrote:
[violet] wrote:No. Neither is publishing a dispatch, as it only generates a Happenings line in your own nation.

Does this also apply if a dispatch is pinned in a region?
It is clear that pinning can only be done manually, as it generates a regional happening, but can a script automatically edit already pinned dispatches?

Probably, because editing a dispatch doesn't generate a happenings line anywhere.
INTP, -4.25 Economic Left/Right, -4.1 Social Libertarian/Authoritarian, tastes like chicken.
I do use NS stats, thank you very much.
Funny Quotes
Pie charts for industries
Request an Embassy

User avatar
HMS Unicorn
Spokesperson
 
Posts: 199
Founded: Jun 29, 2005
Ex-Nation

Postby HMS Unicorn » Sat May 20, 2017 10:36 am

[violet] wrote:
Roavin wrote:6. Is editing a dispatch a restricted action?

No. Neither is publishing a dispatch, as it only generates a Happenings line in your own nation.

Is this accurate?

When I filed a GHR to ask about this a while ago, I was told that editing your own dispatches is not a restricted action, but generating new dispatches is.

The reason is that new dispatches create notifications for nations mentioned in the dispatch. Editing a pre-existing dispatch on the other hand does not create any new nation notifications, even if the edited text mentions nations that were not previously mentioned.

EDIT: The post by Ballotonia that Leppikania references above also contradicts the GHR response (as well as [violet]'s response). Ballotonia says that editing dispatches is a restricted action, whereas [violet] and the GHR response say the opposite.

My GHR was responded to on July 3rd, and therefore post-dates Ballotonia's post. The GHR number is "#174279", filed by HMS Unicorn.

In my GHR, I argued at length about why editing a dispatch should not be considered a restricted action (e.g., editing a dispatch your nation owns is analogous to editing your own nation's page by automatically answering issues, and the latter is not a restricted action).

Can you clarify please?
Last edited by HMS Unicorn on Sat May 20, 2017 10:49 am, edited 5 times in total.

User avatar
Roavin
Admin
 
Posts: 1777
Founded: Apr 07, 2016
Democratic Socialists

Postby Roavin » Sat May 20, 2017 2:22 pm

Thanks for the answers! The answer to #2 is a huge relief.

[violet] wrote:
Roavin wrote:1. Are two simultaneous non-API non-restricted queries based on a single user action permissible?

Yep. Only restricted actions cannot run simultaneously.


I should clarify - the question was phrased poorly. It wasn't necessarily about whether they are allowed to be requested in parallel, but rather if one of those requests would count as scraping (thereby being rate limited) instead of a click, since the user typically couldn't click one button to pull both at the same time on the main site.
Helpful Resources: One Stop Rules Shop | API documentation | NS Coders Discord
About me: Longest serving Prime Minister in TSP | Former First Warden of TGW | aka Curious Observations

Feel free to TG me, but not about moderation matters.

User avatar
Chingis
Events Manager
 
Posts: 561
Founded: Apr 04, 2016
Inoffensive Centrist Democracy

Postby Chingis » Sun May 21, 2017 12:25 am

Following up on Roavin's second question, if two people are running the same script at once, albeit with different UA's, but the scripts don't communicate between each other will it be considered as a "sneaky way of exceeding the rate limit", or is it fine as they're completely separate from each other, with different UA's.

PS: That was a really long run-on sentence, sorry if it doesn't make sense
Last edited by Chingis on Sun May 21, 2017 12:26 am, edited 1 time in total.
1 John 2:1-2 My dear children, I write this to you so that you will not sin. But if anybody does sin, we have an advocate with the Father - Jesus Christ, the Righteous One. He is the atoning sacrifice for our sins, and not only for ours but also for the sins of the whole world.

Matthew 5:43-44 You have heard that it was said, ‘Love your neighbor and hate your enemy.’ But I tell you, love your enemies and pray for those who persecute you.

User avatar
[violet]
Executive Director
 
Posts: 16205
Founded: Antiquity

Postby [violet] » Sun May 21, 2017 5:16 pm

Leppikania wrote:
[violet] wrote:No. Neither is publishing a dispatch, as it only generates a Happenings line in your own nation.

I asked this question previously, and Ballotonia said it was a restricted action. Could you perhaps discuss that?

I'd forgotten that ruling, sorry, and that stands, but I'll raise it with the team to try to reconcile it.

Leppikania wrote:And also, publishing a dispatch could generate a happenings line on someone else's notices page.

By notifying mentioned nations? Notices aren't Happenings lines, so I don't think that's directly relevant, but as mentioned above, I need to discuss with the team first.

Roavin wrote:Thanks for the answers! The answer to #2 is a huge relief.

Note that your UserAgent should accurately identify the user. For example, if you have a tool that you make available for other people to run for their own regions, and what they do with it thereafter has nothing to do with you, make sure their name is going into the UserAgent, too. If we just see a ton of requests all from "Roavin" our first thought will be that they're all you personally.

Roavin wrote:
[violet] wrote:Yep. Only restricted actions cannot run simultaneously.


I should clarify - the question was phrased poorly. It wasn't necessarily about whether they are allowed to be requested in parallel, but rather if one of those requests would count as scraping (thereby being rate limited) instead of a click, since the user typically couldn't click one button to pull both at the same time on the main site.

Yes, most certainly. Very possibly both requests would be considered scraping, and subject to rate limiting rules, if the tool is constructing requests itself rather than just modifying the page.

Chingis wrote:Following up on Roavin's second question, if two people are running the same script at once, albeit with different UA's, but the scripts don't communicate between each other will it be considered as a "sneaky way of exceeding the rate limit", or is it fine as they're completely separate from each other, with different UA's.

That's fine if they are genuinely separate from each other, and not actually sneakily trying to exceed the rate limit, e.g. by pooling their results at the end.

User avatar
Roavin
Admin
 
Posts: 1777
Founded: Apr 07, 2016
Democratic Socialists

Postby Roavin » Sun May 21, 2017 6:38 pm

[violet] wrote:
Roavin wrote:Thanks for the answers! The answer to #2 is a huge relief.

Note that your UserAgent should accurately identify the user. For example, if you have a tool that you make available for other people to run for their own regions, and what they do with it thereafter has nothing to do with you, make sure their name is going into the UserAgent, too. If we just see a ton of requests all from "Roavin" our first thought will be that they're all you personally.


Okay, that'd be the case now :P but I can change that. How about something like:

"Red Discord Bot, operated by Roavin - Server: <ServerName> - Requested by: <Name>"

So, if I were to request something in the Wardens, it would send:
"Red Discord Bot, operated by Roavin - Server: The Grey Wardens - Requested by: Roavin#5410"

Meawhile, if Cormac requests something on the general Gameplay Discord, it'd be:
"Red Discord Bot, operated by Roavin - Server: NSGP - Requested by: Cormac#9408"

Or, any other syntax/information as you want it. Note that I don't know the NS nation names of the people requesting via Discord, only their Discord IDs. Usually they match up, however.

[violet] wrote:
Roavin wrote:
I should clarify - the question was phrased poorly. It wasn't necessarily about whether they are allowed to be requested in parallel, but rather if one of those requests would count as scraping (thereby being rate limited) instead of a click, since the user typically couldn't click one button to pull both at the same time on the main site.

Yes, most certainly. Very possibly both requests would be considered scraping, and subject to rate limiting rules, if the tool is constructing requests itself rather than just modifying the page.


Oh, okay, yeah then that's a bit different. The tool I'm imagining would have a button; clicking it would make two non-API requests and display the results from both interleaved/combined. I would add the "userclick" parameter to both requests. In this scenario, would none, one, or both of those requests be subject to the non-API ratelimit?
Last edited by Roavin on Sun May 21, 2017 6:38 pm, edited 1 time in total.
Helpful Resources: One Stop Rules Shop | API documentation | NS Coders Discord
About me: Longest serving Prime Minister in TSP | Former First Warden of TGW | aka Curious Observations

Feel free to TG me, but not about moderation matters.

User avatar
[violet]
Executive Director
 
Posts: 16205
Founded: Antiquity

Postby [violet] » Sun May 21, 2017 8:06 pm

Roavin wrote:Or, any other syntax/information as you want it.

The idea is to accurately represent what's happening, i.e. who's running the script. Is it a service run by you and used by other people? If so, you are the user. If, however, it's an independent tool that you wrote and handed out and now other people are using for their own ends, they are the user.

I don't use Discord, but I think it's like IRC? In which case the user would be whoever is in charge of that bot, whether that's you, or the channel/server operator. Random Discord users wouldn't be considered the API user; it's unreasonable to expect them to be aware of and follow NS API rules if they're just showing up in a Discord and typing queries. In that scenario, they're more like visitors to a third-party website, and whoever is running the site is the API user.

Roavin wrote:The tool I'm imagining would have a button; clicking it would make two non-API requests and display the results from both interleaved/combined. I would add the "userclick" parameter to both requests. In this scenario, would none, one, or both of those requests be subject to the non-API ratelimit?

The question for me is whether the tool is enhancing nation display pages by adding info about a second nation, in which case only the second request could be considered to be generated by the tool, or doing something entirely new, in which case both requests are tool-generated. So it depends on whether the button is more like "click to view a nation and compare yourself to it" or more like "enter two nations and view our analysis."

User avatar
HMS Unicorn
Spokesperson
 
Posts: 199
Founded: Jun 29, 2005
Ex-Nation

Postby HMS Unicorn » Tue May 23, 2017 7:48 pm

[violet] wrote:
Leppikania wrote:I asked this question previously, and Ballotonia said it was a restricted action. Could you perhaps discuss that?

I'd forgotten that ruling, sorry, and that stands, but I'll raise it with the team to try to reconcile it.

[violet], can you clarify please: Do both parts of Ballotonia's ruling (about publishing and also about editing dispatches) stand? Or just the part about publishing dispatches that Leppikania referenced?

As I mentioned above, if Ballotonia's ruling about editing dispatches stands, then it also contradicts the response to a GHR I had sent about this exact issue - a GHR that dates after Ballotonia's ruling.

User avatar
Eluvatar
Director of Technology
 
Posts: 3086
Founded: Mar 31, 2006
New York Times Democracy

Postby Eluvatar » Tue May 23, 2017 7:53 pm

HMS Unicorn wrote:
[violet] wrote:I'd forgotten that ruling, sorry, and that stands, but I'll raise it with the team to try to reconcile it.

[violet], can you clarify please: Do both parts of Ballotonia's ruling (about publishing and also about editing dispatches) stand? Or just the part about publishing dispatches that Leppikania referenced?

As I mentioned above, if Ballotonia's ruling about editing dispatches stands, then it also contradicts the response to a GHR I had sent about this exact issue - a GHR that dates after Ballotonia's ruling.


The standing ruling is that posting a dispatch is a restricted action but editing one is not.
To Serve and Protect: UDL

Eluvatar - Taijitu member

User avatar
[violet]
Executive Director
 
Posts: 16205
Founded: Antiquity

Postby [violet] » Tue May 23, 2017 8:31 pm

HMS Unicorn wrote:if Ballotonia's ruling about editing dispatches stands, then it also contradicts the response to a GHR I had sent about this exact issue - a GHR that dates after Ballotonia's ruling.

Yes, we're aware of this, and we'll reconcile it.

User avatar
Eluvatar
Director of Technology
 
Posts: 3086
Founded: Mar 31, 2006
New York Times Democracy

Postby Eluvatar » Tue May 23, 2017 9:06 pm

(In case anyone's feeling worried or confused: we're aware there's an ambiguity in provided rulings and are working to resolve it. In the meanwhile, we will not be penalizing anyone who abides in good faith by rulings given.)
To Serve and Protect: UDL

Eluvatar - Taijitu member

User avatar
Ballotonia
Senior Admin
 
Posts: 5494
Founded: Antiquity
Liberal Democratic Socialists

Postby Ballotonia » Wed May 24, 2017 12:30 am

Greetings,

Reconciliation is straightforward: I made a major error in that post, confusing editing and posting a dispatch. It's the ruling for posting dispatches, but written for editing, while they're two totally different things. In doing so I contradicted my own prior postings, the ruling and explanation I handed to other admins/mods, and the rulings I sent out in private messages to other players. Yikes. Please accept my apologies for the caused confusion. The ruling in that post is retracted (I edited it to show the retraction).

Ballotonia
"Een volk dat voor tirannen zwicht zal meer dan lijf en goed verliezen, dan dooft het licht…" -- H.M. van Randwijk

User avatar
Roavin
Admin
 
Posts: 1777
Founded: Apr 07, 2016
Democratic Socialists

Postby Roavin » Wed May 24, 2017 5:19 am

[violet] wrote:
Roavin wrote:Or, any other syntax/information as you want it.

The idea is to accurately represent what's happening, i.e. who's running the script. Is it a service run by you and used by other people? If so, you are the user. If, however, it's an independent tool that you wrote and handed out and now other people are using for their own ends, they are the user.

I don't use Discord, but I think it's like IRC? In which case the user would be whoever is in charge of that bot, whether that's you, or the channel/server operator. Random Discord users wouldn't be considered the API user; it's unreasonable to expect them to be aware of and follow NS API rules if they're just showing up in a Discord and typing queries. In that scenario, they're more like visitors to a third-party website, and whoever is running the site is the API user.


Yeah, it's like IRC in that regard. Okay, they already report as me for now so no need for a change there. Ill mak'e the adjustments so that they identify me as operator for now; a few instances are only run but not operated by me and I suppose within the next few weeks I'll have them modified to report the actual operator, and I'll work on a different technical solution to make sure I don't inadvertently break the limit (probably using Elu's Trawler running on my cloud server that I route everything, including the things on my local computer, through).

[violet] wrote:
Roavin wrote:The tool I'm imagining would have a button; clicking it would make two non-API requests and display the results from both interleaved/combined. I would add the "userclick" parameter to both requests. In this scenario, would none, one, or both of those requests be subject to the non-API ratelimit?

The question for me is whether the tool is enhancing nation display pages by adding info about a second nation, in which case only the second request could be considered to be generated by the tool, or doing something entirely new, in which case both requests are tool-generated. So it depends on whether the button is more like "click to view a nation and compare yourself to it" or more like "enter two nations and view our analysis."


Hmm, this ruling is problematic. Would this also apply for clicks making a single request, then? That is, if a page or tool that is not based on modifications to an existing page (other than page=blank) makes a call to NS, then the rate limit applies even if it's a single request issued on a user click? If this is the case, then wouldn't a variety of existing things, including NS++ and pretty much any NS Android/iOS app, be in violation of script rules?

--

Oh, and in general, thanks for your patience answering my silly questions :)
Helpful Resources: One Stop Rules Shop | API documentation | NS Coders Discord
About me: Longest serving Prime Minister in TSP | Former First Warden of TGW | aka Curious Observations

Feel free to TG me, but not about moderation matters.

User avatar
[violet]
Executive Director
 
Posts: 16205
Founded: Antiquity

Postby [violet] » Wed May 24, 2017 4:50 pm

Roavin wrote:
[violet] wrote:The question for me is whether the tool is enhancing nation display pages by adding info about a second nation, in which case only the second request could be considered to be generated by the tool, or doing something entirely new, in which case both requests are tool-generated.


Hmm, this ruling is problematic. Would this also apply for clicks making a single request, then? That is, if a page or tool that is not based on modifications to an existing page (other than page=blank) makes a call to NS, then the rate limit applies even if it's a single request issued on a user click? If this is the case, then wouldn't a variety of existing things, including NS++ and pretty much any NS Android/iOS app, be in violation of script rules?

No, since the bulk of what they do is (a) reformat regular page requests made by the user, and (b) supplement them with information retrieved from the API. Neither of those things are governed by Script Rules rate limits. The 10-per-minute rate limit applies only when scripts are making their own requests and scraping HTML pages (not the API).

User avatar
Roavin
Admin
 
Posts: 1777
Founded: Apr 07, 2016
Democratic Socialists

Postby Roavin » Wed May 24, 2017 6:32 pm

Well, in the scenario I mentioned, these would also be "user requests" in the sense that they are initiated by the user (though through a tool-generated button, not an actual NS link) and show a reformatted version of them (for example, the graphs side-by-side). Another similar scenario would be a page that interleaves the happenings from the Dossier and Region Dossier activity feeds (since that query can't be made through the API). To apply the latter scenario to a single request, what about a page that pulls the happenings AJAX feed 10 times a minute but can be manually refreshed by the user?
Helpful Resources: One Stop Rules Shop | API documentation | NS Coders Discord
About me: Longest serving Prime Minister in TSP | Former First Warden of TGW | aka Curious Observations

Feel free to TG me, but not about moderation matters.

User avatar
[violet]
Executive Director
 
Posts: 16205
Founded: Antiquity

Postby [violet] » Wed May 24, 2017 7:02 pm

I think it's useful to take a step back here.

You will inevitably encounter awkward problems with ratelimits and restrictions if you build a tool that needs to scrape a lot of HTML pages instead of using the API, because we deliberately discourage this. I've mentioned publicly before that I'd like to banish scripts & bots from the HTML site altogether, so any machine-generated request would have to use the API. The API doesn't provide 100% coverage, though, so what we have today is a situation where bots are allowed on the HTML site under a very tight ratelimit, while I encourage tool authors to post in the API thread to ask for stuff they want added.

Have you tried that? Because it would be a much better use of your time if you can get something added to the API and enjoy the 10X higher ratelimit rather than build an HTML-based tool where you have to worry about whether this or that individual scraping request gets counted.

User avatar
Eluvatar
Director of Technology
 
Posts: 3086
Founded: Mar 31, 2006
New York Times Democracy

Postby Eluvatar » Wed May 24, 2017 9:12 pm

I think that Roavin may be thinking about the one thing which we deliberately don't have in the API but do have in the HTML: the last 28 seconds worth of world happenings.
To Serve and Protect: UDL

Eluvatar - Taijitu member

User avatar
[violet]
Executive Director
 
Posts: 16205
Founded: Antiquity

Postby [violet] » Wed May 24, 2017 10:14 pm

Well, those are specifically delayed for bots. Obviously I have no advice if your bot is scraping HTML just so it can avoid the API rules for bots.

User avatar
Roavin
Admin
 
Posts: 1777
Founded: Apr 07, 2016
Democratic Socialists

Postby Roavin » Thu May 25, 2017 7:12 am

Discouraging HTML scraping in favor of the API is absolutely understandable and reasonable. Now, as this applies to the two specific scenarios I mentioned:

On the nation comparing page: That could be done with the API as is, and from a technical standpoint that would even be the much cleaner solution. Only thing is I'd have to do the graphs myself - the ones on NS are quite pretty and those would be nice to emulate :P

On the combined happenings: Elu is correct, it's about those latest 28 seconds of happenings - BUT it's not about processing, but presentation. Now, the 28 second delay in the API is absolutely understandable and perfectly reasonable, and for tool use there shouldn't be a need to have the latest 28 seconds. This isn't me trying to veer into Predator-territory. This isn't about processing those by the tool, but just for presenting them to the user in ways that the standard activity page reasonably can't. For example, I mentioned combining the regular dossier and region dossier feeds into one. Another good use case would be a tight filter on world happenings plus a wider filter on a specific region's happenings feed. The standard activity feed, as I understood it from briefly perusing the JS code, has a 1.5 second rate which gets delayed over time due to the animations. That already can't be legally emulated, though every 6 seconds would be absolutely sufficient for these purposes. However when multiple feeds are combined, that quickly adds up - two feeds, and we're looking at 12 seconds. Three feeds, 18 seconds, and it quickly adds up. That's why the hope was that this could be reasonably worked around by having the user manually refresh when needed.
Helpful Resources: One Stop Rules Shop | API documentation | NS Coders Discord
About me: Longest serving Prime Minister in TSP | Former First Warden of TGW | aka Curious Observations

Feel free to TG me, but not about moderation matters.

User avatar
Melqart
Attaché
 
Posts: 73
Founded: May 22, 2017
Ex-Nation

Postby Melqart » Thu May 25, 2017 1:11 pm

Another question:

Is API a slow(er) bot or manual work?

If it is manual work, what is the difference between API and templates?
Last edited by Melqart on Thu May 25, 2017 1:11 pm, edited 1 time in total.

User avatar
Phydios
Minister
 
Posts: 2569
Founded: Dec 06, 2014
Left-Leaning College State

Postby Phydios » Thu May 25, 2017 1:45 pm

Melqart wrote:Another question:

Is API a slow(er) bot or manual work?

If it is manual work, what is the difference between API and templates?

The API is an interface. It is a way of accessing site data through computer programs that you or someone else have written.

https://www.nationstates.net/pages/api.html
Last edited by Phydios on Thu May 25, 2017 1:47 pm, edited 1 time in total.
If you claim to be religious but don’t control your tongue, you are fooling yourself, and your religion is worthless. Pure and genuine religion in the sight of God the Father means caring for orphans and widows in their distress and refusing to let the world corrupt you. | Not everyone who calls out to me, ‘Lord! Lord!’ will enter the Kingdom of Heaven. Only those who actually do the will of my Father in heaven will enter. On judgment day many will say to me, ‘Lord! Lord! We prophesied in your name and cast out demons in your name and performed many miracles in your name.’ But I will reply, ‘I never knew you. Get away from me, you who break God’s laws.’
James 1:26-27, Matthew 7:21-23

User avatar
[violet]
Executive Director
 
Posts: 16205
Founded: Antiquity

Postby [violet] » Thu May 25, 2017 4:16 pm

Roavin wrote:This isn't about processing those by the tool, but just for presenting them to the user in ways that the standard activity page reasonably can't.

There is a line between tools that are purely about presentation and tools that initiate requests on their own. But what you're describing sounds like it falls firmly into the latter category, because it's not merely restyling a page or adding buttons, but actually fetching data on its own. That's very close to a classic HTML scraper, and I think it's appropriately bound by those rules.

The 28-second delay on API Happenings exists because it was felt that otherwise tools would be created that delivered a significant gameplay advantage, which would compel everyone to use them in order to remain competitive. I have no opinion on whether that delay is correct or too long or too short; I don't know the area very well. But that's why it's there, and there isn't supposed to be a convenient way to circumvent it.

To achieve the outcome you want, I see three options:

(1) Use the API to pull desired feeds and combine them however you like.

(2) Make a case for new view types to be added to the standard Activity page.

(3) Create a very limited HTML-fetching tool that will never initiate more than 10 requests per minute.

#3 is obviously discouraged, and far more dangerous, since on the API, if you accidentally break the ratelimit, you just get automatically locked out for 15 minutes, while if you break HTML scraping ratelimits, we may take action against your account. It's also fully your responsibility to stay within the rules. I see NS admin's job as being to make the rules clear, but not to inspect a specific tool's code or methodology for pre-approval. So while I'm happy to discuss the rules' principles, I won't step through projects with tool authors to help them figure out how to sail one inch this side of the rules on HTML scraping when they should really just use the API. Although Eluvatar is more generous with that kind of thing.

Next

Advertisement

Remove ads

Return to Technical

Who is online

Users browsing this forum: Amiria Grande, Cerespasia, CONNN, Falloa, Grandocantorica, Improper Classifications, Liravia, Mafiocracy, New Fernia, Novimor, Ostrovskiy, Polgaria, Radicalania, Shearoa, The Southern Dependencies, The Terren Dominion, The Tumbran Northlands

Advertisement

Remove ads