32
Addressing kbin.earth stability issues (kbin.earth)
submitted 4 months ago by to c/kbinEarth@kbin.earth
17 comments fedilink hide all child comments
 

As I'm sure y'all have noticed, the server has been having stability issues the past week. Every once in a while, the server seems to "crash"; it will time out on all network requests for about 5 minutes, happening several times within an hour. Additionally, there have been several times over the past week when the server queues got backed up, resulting in incoming and outgoing federation traffic being delayed.

At first, I assumed the server hardware had simply become insufficient to handle all of the network and federation traffic we have been receiving, potentially due to a DDOS attack. I was honestly starting to think I would need to upgrade the server hardware, which is ridiculous, as the VPS is already quite powerful and is certainly expensive enough.

Luckily, @green_copper helped with debugging and noticed the server timeouts only occur when the microblog section is viewed (and sometimes when the combined section is viewed). So rather than a lack of hardware being the issue, I believe it actually has to do with the recent Mbin upgrade. My guess is that an SQL query was changed in one of the commits between Mbin v1.9.1 and v1.10.0, which resulted in a widely inefficient query, causing the server to freeze up and time out on requests.

I have and will be investigating the issue this week. I will start by analyzing which database queries are causing the timeouts, but if needed, roll back to Mbin v1.9.1 and try to figure out which specific Mbin commit introduced the bug. Additionally, I may have access restricted only to signed-in users while I'm working on this.

In the meantime, if you guys can refrain from viewing microblogs (or combined view), that would keep the server accessible for everyone and help me work on it faster, as I can't do anything while the server is frozen. If you want any realtime updates on the server issue (or anything else related to kbin.earth), feel free to join the Matrix room.

Thanks for sticking around and for your patience!

all 17 comments

sorted by: hot top controversial new old
[–] 14 points 4 months ago (1 child)

Thanks for sticking around and for your patience!

And thank you for dedicating your time and money providing us with a nice corner of the internet ^^

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
    • [–] 6 points 4 months ago

    You rock @jwr1@kbin.earth

  • permalink
  • fedilink
  • source
    •
    [–] 6 points 4 months ago (1 child)

    Good luck! If it's truly a SQL issue, it seems you've got your work cut out for you. There's some QB query builders, conditional string concatenation, big preformatted blocks of nested subqueries, the works. Will likely be hard to test.

    I do wonder if it has to do with the move to cursors for pagination, there's quite a number of changes related to that.

    https://github.com/MbinOrg/mbin/pull/1985 https://github.com/MbinOrg/mbin/pull/2041

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
    • [–] 6 points 4 months ago

    While attempting to find where to change my default home page view off of the combined view, I mis-clicked and pressed the home button and then the server crashed. So I apologise for being a moron. Thank you for all your work with the server

  • permalink
  • fedilink
  • source
    •
    [–] 5 points 4 months ago

    Went in and switched my default view off combined. Good luck! This sounds like a complicated issue, but I hope it actually turns out to be simple

  • permalink
  • fedilink
  • source
    •
    [–] 3 points 4 months ago* (1 child)

    @BentiGorlich@gehirneimer.de, maybe related to the RSS and php-fpm process situations at The/Brain/Bin?

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
  • [–] 2 points 4 months ago

    that was actually mostly a pg-bouncer. I have no idea why it is happening though...

    I just have a script that checks for this line and restarts php-fpm if necessary:

    #!/bin/bash

lineCount=`tail -n 1 /var/log/postgresql/pgbouncer.log | grep query_wait_timeout | wc -l`

if [ $lineCount -gt 0 ]; then
  systemctl restart php8.3-fpm.service
  logger "Restarting php because pgbouncer has query_wait_timeout messages"
else
  echo "No"
fi
  • permalink
  • fedilink
  • source
  • parent
    • [–] 3 points 4 months ago

    The prime24/nordpublications jackasses have switched tactics and are creating accounts still without posting their vacuous rubbish. Watch new accounts, in not saying it's them but they're highly suspect, imo.

  • permalink
  • fedilink
  • source
    •
    [–] 1 point 3 months ago (2 children)

    Noticed the banner isn't up anymore. Are microblogs usable again or should we still refrain until you post an update?

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
    • [–] 1 point 4 months ago (3 children)

    Just wondering if you've considered Anubis rather than cloudflare?

  • permalink
  • fedilink
  • source
  • hideshow 3 child comments
    •
  • [–] [S] 2 points 4 months ago (2 children)

    I have considered it in the past, but there are a number of reasons I am still using Cloudflare.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •