[–] [S] 3 points 1 year ago (4 children)

We don't really talk, but I think they all struggle with this in some way. Many instances use Cloudflare for example.

The bots seem to be trying to pull many random posts (e.g. posts that were federated across from other instances)

My guess is they're trying to vacuum up the content. It could also be a low-level DoS attack but it feels too light and inefficient for that.

  • source
  • parent
  • context
  •  

    Over the past few weeks, our little site has been slammed by web scraping bots.

    There is a lot of headroom on the server so the increased load has not led to any noticeable degradation, but the numbers are pretty wild. Here are some stats for the past 2 weeks:

    We may be popular, but we're not THAT popular.

    The typical solution to something like this is to embrace services like those provided by Cloudflare. I absolutely do NOT want to do this, unlike many of our Fediverse peers. I believe surrendering autonomy and privacy in exchange for security is incompatible with what we're trying to do here.

    Instead, I have been working on a PoW challenge system, with the inspiration coming from projects like Anubis (https://github.com/TecharoHQ/anubis)

    The idea is simple. Clients need to solve a set of simple cryptographic challenges before they are allowed access to the site content. This rules out the vast majority of simple scrapers, and makes it computationally expensive for the more sophisticated ones. I'm calling it Tollbat because why not.

    The trade-off is you will have noticed a Tollbat challenge screen before accessing HC. This could mean a 5-10 second load time every so often. I will continue to tune it to make it as light as possible for real users.

    Federation, 3rd party apps like Jerboa and legitimate bots are not affected. So far things seem to be going well, but let me know if you notice any weird behavior.

    The load reduction is significant to say the least. Here are more graphs for those who like them - see if you can spot when Tollbat was turned on:

    Feel free to ask any questions, I'm happy to answer them.

    [–] [S] 3 points 1 year ago (1 child)

    Not related, I didn't think anyone was still using ok0.org. When I last had a look, it was bots finding new ways to spam just to be annoying, Asian hookers peddling their wares and crypto bros sharing their latest musings.

    I guess it doesn't help that I haven't used it in ages, but it just felt like nothing changed in the two years I've been running it. It felt like a waste so I pulled the plug on it, but I can be persuaded otherwise if you think it's worthwhile.

  • source
  • parent
  • context
  •  

    This is a heads up that there will be at least two outages over the next few days, each lasting a few hours.

    I need to move some physical hardware around, and there's no way to do that without unplugging it.

    If you notice that HC isn't working, there isn't any reason to worry, it's coming back.

    [–] 5 points 1 year ago* (1 child)

    Not at all, we do not disallow any kind of critical speech, unlike the vast majority of Lemmy instances. The TOS are somewhat broad as it's hard to be specific with these things and discretion always plays a part.

    In general, criticism, debate and satire are perfectly fine. Discrimination, personal attacks and insults are not. To give a concrete example, you can say that you don't like religion X for Y reasons as much as you like. You can't say that all people of X religion are idiots and you hope they die.

    It's no different than speech about anything else.

    Another thing that is relatively unique about our instance is we give communities the right to moderate themselves according to their own rules, so long as they stick to our TOS. This means that you could, for example, start a community dedicated to praising Cthulhu and remove all comments that are even vaguely critical of him from your community, without having the HC admin try to police what you should or shouldn't allow.

    These things make us somewhat unpopular in the Lemmy ecosystem where censorship along political lines is seen as desirable, so long as it favors the left. HC is fairly well known, despite our small size, specifically because we refuse to act like the thought police.

  • source
  • [–] 2 points 1 year ago* (1 child)

    This right here sums it up nicely:

    The whole shtick with Fediseer is to pressure instances into falling in line under threat of defederation. It's reasonably effective, most small instances would struggle when cut off from chunks of the federated network they're trying to be a part of.

    I also think most Admins probably hope to get some money coming in through donations, and you need to be reasonably vanilla and attract new people for that to work.

    We really don't need donations to keep running, in fact we don't even ask. I'll be damned if I let some random Internet dude with an over-inflated sense of self-importance tell me who I can or can't allow on the instance I run as a hobby, as to not offend his political sensibilities.

    @realcaseyrollins@narwhal.city I meant this as a reply to your post but failed.

  • source