Are you calling Firefox a scraper then?
I’m sure there’s a way to wire it up that way.
Yes, you simply turn off images. Then Firefox is a scraper from the server standpoint. I use FF that way when operating under a bandwidth quota. Scraping a page is far less of a burden on the server than fetching the whole page (images, js, and all) for GUI rendering.
It’s bound to have a Turing complete language in it somewhere
Turing completeness is wholly irrelevant. A fetch operation does not absolutely need a turing complete language. But you will struggle to render the content for a graphical presentation without a turing complete language.
From a practical standpoint there’s a spectrum of activities that approximately go from Browsing to Scraping to DDoS in that order that really only differ in terms of scale.
Not “in that order”. Browsing graphically causes every image to be fetched. It’s much heavier than the HTML text that a scraper pulls. 10,000 gui users each fetching a page will strain your webserver faster than 10,000 html text pages being scraped.
Non-graphical users suffer from marginalization driven by the shitty attitude that “bots are bad and humans should alter their behavior and fetch images to look less like a bot as we don’t give a shit about oppressing bots.”
It sounds like what you’re describing is just timeshifted browsing. I doubt anyone notices unless you’re really describing what you’re trying to do incorrectly.
Timeshifted browsing is purpose of HTtrack. An AI operation doesn’t use HTtrack which is designed to organize the browsing session into interactive projects which launch a local GUI browser for offline viewing. An AI op has zero interest in that. A scraping operation uses curl, wget, beautiful soup/python, etc.
until WHOOM! an AI company scrapes their whole site and their hosting cost jumps that month, then the next month and the next as well because companies that would destroy copies of rare books to feed to their training don’t care.
It’s this kind of commentary that distorts public perception.
Scraper bots themselves serve many users, often more than the original site could afford to host. A scraper bot that grabs just raw text builds a database and makes the information more useful to more people than the original site. E.g. instead of each traveler manually visiting the website of every single airline (a costly image-rich bloat high in CO₂ emissions), they can go to a site that has scraped Ryanair and others. The aggregated text info is more useful. Consumers then only have to visit /one/ airline website after seeing all the info side-by-side. This is much less strain on resources.
You’ll often hear/read “follow” used by itself to imply “follow the rules,” in this case “follow the rules specified by robots.txt.”
That’s what I was saying. Hence why the claim that HTtrack follows locations listed in robots.txt is nonsense.
From a technical standpoint an LLM recommends trying this code snippet to make httrack ignore the robots.txt file:
I appreciate the tip. Indeed -s0 is what I worked out that I needed. You seem bent against AI so it’s a bit unexpected that you would be using it. I avoid AI because I generally boycott the corporations pushing the LLMs. But I’m not generally opposed to LLM tech. I have a local language translator and probably need to look into other LLMs that can run locally.