Hello everyone,

I have an issue with my desktop, which I really don't understand.

It is a desktop PC, with a Nvidia GPU, which I use for development (of AI, but like the image analysis part, not the slop part). I usually let it always run, and my previous machine was working fine like this.

However, on this machine, I get crashes after either a few days or a few weeks.

These "crashes" are weird : the machine is still up (and I can ping it), however I can't ssh into it, I can't connect to cockpit either, and can't login physically on it (the display is frozen).

The only way to get it back up is to forcefully shut it down, wait a little and then powr it back up; but this isn't ideal, since it interrupts computations etc

The only crashes shown in journalctl are related to irrelevant (I believe) programs : KDEConnect, DrKronqi, etc; nothing right before the crash.

Do you have any idea what could cause this ?

I tried logging temperatures, but nothing abnormal showed up. I ran memtest86+, nothing showed up either. Smart tells me everything is alright for my drives as well.

Right now I'm thinking about changing the alimentation or the motherboard, but frankly I don't know what's going on and it's really puzzling me.

all 13 comments

sorted by: hot top controversial new old
[–] 11 points 6 hours ago (1 child)

Sounds like a long tedious elimination process is ahead.
But since the only errors you get regard KDE, I'd try installing another desktop environment. Afterwards you can generally choose which desktop to log into at the login screen. And when it's resolved whether or not KDE is the problem, you can easily switch back if you want. So it's very easy to try.
But each elimination will take 5 days!! If the computer behaves the same each time it crashes, it is definitely NOT random. If it was random it would be different things each time, and then it could be RAM or even CPU problems. But this sounds like a software problem.

  • source
  • hideshow 1 child comment
  • [–] [S] 2 points 5 hours ago

    Thank you for your answer ! I'll try it (nothing to lose), but more often than not I don't login to the desktop; but maybe de login screen is enough ? I don't know, I'll try using gnome anyway.

  • source
  • parent
  • [–] 9 points 6 hours ago

    I've had similar issues when I ran out of system memory. It also didn't leave any traces in the journaling program, but I think it did say something about it in dmesg. Not sure though. But since you're probably using the VRAM and not the RAM, maybe this isn't it at all? Just spitballing.

  • source
  • [–] 6 points 5 hours ago*

    I had a similar issue on bazzite and I think it had to do with the Out Of Memory service. This link might help if that's it.

    Disclaimer* if you look at my account I have mentioned that I only joined linux this year and have only distro hopped once so beyond this post I can't really offer much assistance.

  • source
  • [–] 6 points 5 hours ago*

    Well drkronqi is a crash handler, and KDE connect has its fingers in multiple pies? You say theyre not relevant but who do you know when the crash is, if you only notice it when you try to use the PC and its unresponsive? I wouldn't dismiss the timing of these crashes if you can't say exactly when it stopped functioning. They may even by symptoms if not the actual cause.

    I'd start by increasing the logging levels to make them more detailed. You change the kernel and system logs to make them more detailed - how will differ slightly on your distro but essentially you want debug level system logs and lowest severity kernel logs. You can also setup some extra logging such as outputting system memory and vram usage to a dedicated file every second or so. This will.give you more information to hunt for the culprit.

    As others have said, my first thought from your useage and description is a memory issue. That could be RAM or vram - especially with the AI useage. I'd start by logging both to their own files; out of memory can crash without much trace if the logs aren't tracking the right things.

    Also whatever process you are using - your AI tool for example - get that logging too.

  • source
  • [–] 2 points 5 hours ago (2 children)
  • [–] [S] 4 points 5 hours ago* (1 child)

    64G RAM, 8G Swap (i bought it about 2 years ago, and it's worth a lot more now...)

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 1 hour ago

    64GB sounds plenty, but is there perhaps a slow men leak in a process?

    Leave your PC running, run this in a terminal, and leave it:

    watch -n 3600 "date >> logfile.txt; free -m >> logfile.txt"
    

    Its a script that writes the time, and then the memory stats to logfile.txt.

    Let it run until the PC stops responding, and then leave the PC a couple more hours.

    When you reboot, have a look at the logfile, and see if the memory was growing at all. Also see if you get any date stamps after the PC stopped responding, because that might mean it was not fully crashed.

    If it is a memory leak, the next step would be narrowing down which process.

  • source
  • parent
  • [–] 2 points 5 hours ago

    I did have an old laptop with bad SSD that would have similar problem - cannot ssh, but can ping, and desktop quickly gets unresponsive.

    On root device issues kernel remounts root drive to read-only, and that breaks everything, but the machine still works - yet you get no logs because /var/log is also read-only.

    I did some googling and it seems SSH requires writable /tmp, so you might setup some in-memory one, if you haven’t already - you could check if your machine uses tmpfs (in-memory tmp) to reject this hypothesis.

  • source
  • [–] 1 point 5 hours ago

    i am going to follow this as i am having a similar issue (on multiple distros even and with nothing helpful in the journal either). i went as far as to replace the entire PC, suspecting a hardware issue (it was still under warranty) but the problem remained the same.
    all i could find on the internet is it may be related to a combination of wayland and KDE plasma iirc. i deactivated stuff recommended by users and tried other suggestions but nothing ever resolved the issue. i have a feeling it has to do with certain updates after which the issue ebbs and flows but this might just be because the problem only occurs randomly in the first place, so this is an entirely "vibes based observation".
    my issue is definitely not an out of memory issue as i had btop open when it crashed and it showed memory usage of ~60%.

  • source
  • [–] 1 point 2 hours ago

    without knowing what you are running and your hardware, it's hard to say. The symptoms sound similar to when I was running AI on a small server with 32gb ram only, and I hot oom, I'd be able to ping it but nothing else would work and would require a hard reset.

  • source
  • [–] 1 point 2 hours ago

    Is it a Ryzen CPU? Do you have data fabric sync flood errors in the journal?

  • source
  • [–] 0 points 5 hours ago

    Have you done a memtest? https://memtest.org/

  • source