18
[Cancelled] Maintenance: 01:00~01:30 UTC (Server Resource Increase) (reddthat.com)
submitted 1 year ago by to c/reddthat@reddthat.com
6 comments fedilink hide all child comments
 

That's 2 hours from this post.

At 01:00 UTC we will be having a 15-30 minute maintenance due to a server upgrade. (More storage!)

As a reminder we have a status page on: status.reddthat.com which I use when we have these outages.

(Yes I'm aware of 'old' showing up as down, the uptime checker is blocked by our anti-ai protection for old)

See you on the other side.

So apparently it's not instant and we are in a queue to get upgraded... We are... 65th in the queue. I'll edit this post once we get an update.

you are viewing a single comment's thread
view the rest of the comments
[–] [S] 7 points 1 year ago (3 children)

Unfortunately not. We are in a queue that will happen... Whenever OVH (our server host) lets me know.

The outage/downtimes just now were because an AI scraper just brought down the server. We had ~81000 unique IPs spam our little server. It wasn't a smart AI scraper so I managed to block them and we are back to being up and working.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 1 year ago* (last edited 1 year ago) (2 children)

    Ouch, that's just so annoying. Maybe some sort of human challenge for anyone that's not authenticated whenever the server load is high could help?...

    Edit: unless the scraping is coming from the ActivityPub protocol as a bunch of instances trying to federate (or something? I'm still not familiar with how it all works, besides the payloads looking like a bunch of .json), in which case, my suggestion probably wouldn't apply all that well.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] [S] 3 points 1 year ago (1 child)

    Yeah we have implemented go-away on https://old.reddthat.com/ but only as a test and it's worked well. I haven't made the time to implement it for our main endpoint.

    I also want to do exactly that, only show the challenge for non-logged in users which adds a small bit of complexity but my $job has been flat out recently. So I've only been doing moderation duties rather than server admin duties.

    But I can think of plenty of edge cases that something could happen or affect regular users and I don't want that to happen. Normally the bots are all like the ones we just had, using open proxies/residential proxies and querying either posts or the API directly.

    I'm hoping to carve out some time in the next month to coincide with this maintenance or our server migration to do some solid testing.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 1 year ago

    Hey, if you need any assistance, I happen to be a DevOps engineer. Not sure how much help I could give, and my own $job comes first too as well of course, but I'm sure there's a bit of overlap somewhere where my skills may be of assistance if you need me for something specific and small to be implemented, and I'm quick at the pickup at least.

    I'm also familiar with Docker. Though granted, in CICD (create/build/destroy) scenarios, not in persistent hosting.

  • permalink
  • fedilink
  • source
  • parent