[Q] Legality of Scraping the NS forum

[The Ice States]

Greetings!

Is it legal to send scripts to the forums to scrape a forum page without being logged-in? I'm currently working on a script to scrape HTML from NS forum threads for the purpose of archival, which sends automated requests to the NS forum for the purpose of saving the HTML code while logged-out. I would appreciate confirmation on whether this is fully legal, as I have been advised that scraping the NS forum is frowned upon?

Thanks!

[United Calanworie]

Please note that this is not a ruling.

Most likely, yes. Our rules prohibit sending requests as a logged-in user automatically. So presuming that you will not be doing that, you would have to adhere to no more than ten requests per minute. Obviously though, it's heavily discouraged to scrape the forums. If you're going to persist in this idea, please consider not going anywhere close to the ratelimit.

[Valentine Z]

For clarity sake, I have been doing this for a long while at this point, again while I'm not logged in. I didn't use scripts or anything of that sort, but I do have an URL opener (it's a browser extension) that lets me open multiple URLs at once (and slowing down the computer a bit). Then I have another extension to download webpages as HTML.

In essence, open 500 tabs, then save them as HTML, both requiring my manual input.

Does the rate limit apply for this as well? As in, can I open all 500 tabs at one go, or should I open in small batches? Thanks in advance!

EDIT: Here's the ongoing project of mine for reference: https://www.nationstates.net/page=dispatch/id=1543370

[Racoda]

For clarity sake, I have been doing this for a long while at this point, again while I'm not logged in. I didn't use scripts or anything of that sort, but I do have an URL opener (it's a browser extension) that lets me open multiple URLs at once (and slowing down the computer a bit). Then I have another extension to download webpages as HTML.

In essence, open 500 tabs, then save them as HTML, both requiring my manual input.

Does the rate limit apply for this as well? As in, can I open all 500 tabs at one go, or should I open in small batches? Thank in advance!

EDIT: Here's the ongoing project of mine for reference: https://www.nationstates.net/page=dispatch/id=1543370

According to some earlier questions*, it is a tool and does fall under the rate limit:
viewtopic.php?p=32647607#p32647607

*I'm saying questions, because I'm sure there was another more recent thread on the matter, but I can't find it.

[Imperium Anglorum]

I've tried running requests (the Python package) against the forum circa September this year and get a pile of Cloudflare gobbledygook. Is that intentional or can nothing be done on it?

[Valentine Z]

For clarity sake, I have been doing this for a long while at this point, again while I'm not logged in. I didn't use scripts or anything of that sort, but I do have an URL opener (it's a browser extension) that lets me open multiple URLs at once (and slowing down the computer a bit). Then I have another extension to download webpages as HTML.

In essence, open 500 tabs, then save them as HTML, both requiring my manual input.

Does the rate limit apply for this as well? As in, can I open all 500 tabs at one go, or should I open in small batches? Thank in advance!

EDIT: Here's the ongoing project of mine for reference: https://www.nationstates.net/page=dispatch/id=1543370

According to some earlier questions*, it is a tool and does fall under the rate limit:
viewtopic.php?p=32647607#p32647607

*I'm saying questions, because I'm sure there was another more recent thread on the matter, but I can't find it.

I forgot that I have asked that before in that thread. My bad, then!

For the subsequent archives I'm doing, I'll follow the rate limit. Slowing down will actually help me, in all honesty, because having a lot of tabs loading at the same time does cause a bit of slowdowns.

[Roavin]

Is it legal to send scripts to the forums to scrape a forum page without being logged-in? I'm currently working on a script to scrape HTML from NS forum threads for the purpose of archival, which sends automated requests to the NS forum for the purpose of saving the HTML code while logged-out. I would appreciate confirmation on whether this is fully legal, as I have been advised that scraping the NS forum is frowned upon?

United Calanworie already said the important bits. Beyond that, it might help if you explain what you're trying to achieve — "archival" is sufficiently vague that it can range anywhere from "I want to save a handful of RPs I've been involved in" (which is fine) to "I want to mirror the forum" (dear god, no, that's hundreds of gigabytes of data).

[Valentine Z]

Is it legal to send scripts to the forums to scrape a forum page without being logged-in? I'm currently working on a script to scrape HTML from NS forum threads for the purpose of archival, which sends automated requests to the NS forum for the purpose of saving the HTML code while logged-out. I would appreciate confirmation on whether this is fully legal, as I have been advised that scraping the NS forum is frowned upon?

United Calanworie already said the important bits. Beyond that, it might help if you explain what you're trying to achieve — "archival" is sufficiently vague that it can range anywhere from "I want to save a handful of RPs I've been involved in" (which is fine) to "I want to mirror the forum" (dear god, no, that's hundreds of gigabytes of data).

I can vouch for The Ice States, if I may, since we have talked about this a while back and did a collaboration of sorts.

For all intents and purposes, we are not trying to mirror the entire forum, or even all of a subsection (Forum 7); our aim is to archive noteworthy and effortful threads that have hit 500 pages and are locked, in order to save them as part of F7's history before they are wiped by the 7-day window. So this is an occasional archive because it does take a while to hit 500 pages (even popular threads take months).

I hope this helps!

[Site-]

For clarity sake, I have been doing this for a long while at this point, again while I'm not logged in. I didn't use scripts or anything of that sort, but I do have an URL opener (it's a browser extension) that lets me open multiple URLs at once (and slowing down the computer a bit). Then I have another extension to download webpages as HTML.

In essence, open 500 tabs, then save them as HTML, both requiring my manual input.

Does the rate limit apply for this as well? As in, can I open all 500 tabs at one go, or should I open in small batches? Thanks in advance!

EDIT: Here's the ongoing project of mine for reference: https://www.nationstates.net/page=dispatch/id=1543370

I've been told in private by an admin that usage of a multilink open tool that opens multiple links with one action is illegal before.
This was more then 2 years ago.
In my case the tool was Snap Links Plus.


Edit: Forgot to login my main, this was Vylixan posting .

[The Ice States]

United Calanworie already said the important bits. Beyond that, it might help if you explain what you're trying to achieve — "archival" is sufficiently vague that it can range anywhere from "I want to save a handful of RPs I've been involved in" (which is fine) to "I want to mirror the forum" (dear god, no, that's hundreds of gigabytes of data).

I can vouch for The Ice States, if I may, since we have talked about this a while back and did a collaboration of sorts.

For all intents and purposes, we are not trying to mirror the entire forum, or even all of a subsection (Forum 7); our aim is to archive noteworthy and effortful threads that have hit 500 pages and are locked, in order to save them as part of F7's history before they are wiped by the 7-day window. So this is an occasional archive because it does take a while to hit 500 pages (even popular threads take months).

I hope this helps!

For the record, I can confirm this.

[Roavin]

If you stay below the rate limit, identify your script to the server via User Agent, and only do it occasionally to pull such threads, that should be fine.

[United Calanworie]

My concern would be making sure that you spread out those 500 requests sensibly. I would say that you likely shouldn't do more than one or two requests per minute, as 10 requests a minute would result in nearly an hour of high scripting load for the forums.

[Valentine Z]

Hmm, okay. Dumb question, sorry about that, I just want to make sure that I am clear on this.

What about browser extensions, because for those, there is no way to identify myself as the one using it. Here's the extension in question that I am using, which I drop a bunch of links and open them on my computer: https://github.com/htrinter/Open-Multiple-URLs/

Thanks, and sorry to ask again!

EDIT: Or should I move on using a script that has an user agent instead of using a browser extension? Either way should work for me.

[[violet]]

We ban all bot activity from the forums, but that's mostly just because it's being constantly attacked by them, and I have to deploy aggressive counter-measures to keep things working.

I have no problem with people very slowly scraping relatively small amounts of data for archive, but I can't explicitly support it (because our anti-bot stuff, which I don't want to disable, may block you), and I also don't want to have to deal with well-meaning authors who write bots that hit an unexpected corner case and then start smashing the site at maximum speed, which happens more often than you'd think.