843
lads (lemmy.world)
submitted 1 year ago by to c/programmer_humor@programming.dev
57 comments fedilink hide all child comments
 
you are viewing a single comment's thread
view the rest of the comments
[–] 7 points 1 year ago (5 children)

I just looked at my log for this morning. 23% of my total requests were from the useragent GoogleOther. Other visitors include GPTBot, SemanticScholarBot, and Turnitin. That's the crawlers that are still trying after I've had Anubis on the site for over a month. It was much, much worse before, when they could crawl the site, instead of being blocked.

That doesn't include the bots that lie about being bots. Looking back at an older screenshot of a monitors---I don't have the logs themselves anymore---I seriously doubt I had 43,000 unique visitors using Windows per day in March.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] -1 points 1 year ago* (4 children)

    Why would they request so many times a day the same data if the objective was AI model training. It makes zero sense.

    Also google bots obeys robots.txt so they are easy to manage.

    There may be tons of reasons google is crawling your website. From ad research to any kind of research. The only AI related use I can think of is RAG. But that would take some user requests aways because if the user got the info through the AI google response then they would not enter the website. I suppose that would suck for the website owner, but it won't drastically increase the number of requests.

    But for training I don't see it, there's no need at all to keep constantly scraping the same web for model training.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 6 points 1 year ago* (3 children)

    Like I said, [edit: at one point] Facebook requested my robots.txt multiple times a second. You've not convinced me that bot writers care about efficiency.

    [edit: they've since stopped, possibly because now I give a 404 to anything claiming to be from facebook]

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 3 points 1 year ago (2 children)

    You've not convinced me that bot writers care about efficiency.

    and why should bot writers care about efficiency when what they really care about is time. they'll burn all your resources without regard simply because they're not who's paying

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 4 points 1 year ago (1 child)

    Yep, they'll just burn taxpayer resources (me and my poor servers) because it's not like they pay taxes anyway (assuming they are either a corporation or not based in the same locality as I am).

    There's only one of me and if I'm working on keeping the servers bare minimum functional today I'm not working on making something more awesome for tomorrow. "Linux sysadmin" is only supposed to be up to 30% of my job.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 3 points 1 year ago*

    I mean, I enjoy linux sysadmining, but fighting bots takes time, experimentation, and research, and there's other stuff I should be doing. For example, accessibility updates to our websites. But, accessibility doesn't matter a lick if you can't access the website anyway due to timeouts.

  • permalink
  • fedilink
  • source
  • parent