I lost a raspberry pi for a couple months. Just couldn't find the darn thing. It was working fine serving away.

Turns out it was in the basement. I forgot I moved it to be closer to my tilt Hydrometer.

For a while, the windows machine I was using kept randomly becoming unresponsive (both via network, as well as the keyboard+mouse) and I just could not figure out why. Everytime it did, I'd have to force a shutdown by holding the power button until it shutoff, then press it again to start back up. Sometimes this happened while I wasn't home, so it had to just stay offline until I was.

I got sick of having to do this; so I replaced the power button with a transistor and a RPI. The PI would ping the server every 5min; if it failed to get a response 3 times in a row, it would trigger the transistor for 10sec, release for 3sec, then press and release again; forcing a poweroff then starting the machine again. It'd also write these events to a log file. I think it was around twice a week ish.

Never did figure out what the cause of the lockup was; but it stopped when I replaced windows with Debian.

mouse not pictured

Got my hopes up for nothing!

I hosted shit for about ten years before really understanding how local dns worked.

Imagine a primitive society that discovered literally every major modern scientific achievement, but somehow missed the wheel. That's basically how my first setup went lol.

One day, our internet just stopped working entirely. Our ISP was fine, router good, but no outside connection and I couldn’t access local devices. I have two Unraid servers. I took a quick glance at them, they were powered on, all looked ok. I decide to power cycle everything. Still no internet. Then I realize that I can still access Google from my phone when on WiFi.. hmm. That would mean… an issue with pi-hole!

I begin to try to diagnose from my phone with a hotspot to my computer. I’m getting warmer but some things just aren’t making sense. I go back and stare at the server running pi-hole. Then it fucking hits me.

I have the pretty odd and totally not ideal home network. I have plans to upgrade everything but currently I have two wifi networks tied to the same WAN. One network is run off of access points so I can get Ethernet to odd areas of the property. The Unraid server hosting pi-hole was connected to an AP and the Ethernet cable I used didn’t have a clip, so it just barely unseated itself from it, breaking the connection. The router+pi-hole config meant also breaking the internet if anything happened to pi-hole or that server. I swapped it for a better cable and it all instantly started working.

Spent that weekend sorting out the network so it doesn’t happen again. It was definitely a head-scratcher.

Then you tell people about it on IRC and become known as "the dead mouse guy".

But the maximum nickname length on the IRC server is limited, so you settle for "deadmau5" as your nick.

At my first-ever tech job back in 2000, I was given a little Celeron desktop computer to do my work: mostly writing some ColdFusion and cutting up images in Photoshop. For the most part, the machine worked fine, except whenever I scrolled down on a webpage in Netscape, the box's PC speaker would start screaming. I tried to ignore it, but I was in a small room with two other nerds and it was getting really annoying.

One day, my frustrated colleague decided it was time to investigate. As part of the process, he noticed that there was very little air coming out of the power supply fan port. "Must be a busted fan and that's causing an overheating alarm" he declared. We unplugged the box, popped it onto my desk, and opened it up. The internals were pristine. No dust, nothing. The computer was practically brand new after all.

We were just about to crack open the power supply (not recommended, but we were getting desperate) when my colleague noticed that something was wedged inside the fan... it was the biggest (dead, thankfully) cockroach I have ever seen, at least 10cm long. For context, this was in Vancouver, Canada. Those just don't exist there.

My colleague jammed his screwdriver in there to grind up the cockroach carcass, then plugged the box back in. After a shittone of dessicated cockroach guts spewed out the back, my little computer was operating normally.

As best we could guess, the monster crawled into the fan in Malaysia when as was assembled and somehow got pinned in there, died of starvation, and then stowed away to Canada.

No mouse problem, but I've had to take measures to prevent cat-related downtime. Cats and power-button at the top is not a good combination.

Optical or roller ball?

You don't need a mouse for a server. Just SSH into it, duh.

Not exactly self-hosting, but I've worked at two different academic labs that had unique issues and small budgets.

The first was on top of a mountain at 3km elevation. The low air pressure meant that all hard drives had a lifespan of no more than a year before starting to have issues. This was in the pre-SSD days. The "solution" was to boot from a floppy, never compress data, and have a student return the drives to a low elevation and swap them for the prior ones every six months.

The second was at a place that was using old Datacube MaxVideo hardware that we would never have the budget to replace. It would crash if the room temperature got above about 18C, so we set the thermostat to 15C. Which doesn't sound too bad until you've tried sitting at a desk for 8 hours where everything is 15C and the heat is just slowly leaving your body into the chair, the desk, and the keyboard. And then you walk outside where it is 32C and instantly start sweating because you're still in several layers and fingerless gloves.

Not selfhosting, but I remember standing in a server room with a hands' width of water on the ground (and rising), and hastily shutting down servers where the water was already in the case. It were big towers (Compaq Proliant, if anyone remembers), and luckily the board, drives, and PSU were in the top part of the big tower cases, the drive bays at the bottom were empty.

We got all equipment out safely, and we did not get electrocuted, either. We left with the last equipment when the water level was approaching the knees.

Back in the day of BBS’ my buddy had a wonky modem. If I picked up phone , pursed my lips and did a squeaky sound the modem worked and I could log into BBS.

Fail2ban crashing on 3 seperate days i tried setting it up then one day i did the exact same thing an it worked

Goddamn I hate mice. I just had to tear out the entire contents of a combine cab to get rid of their piss and shit, and fix the wiring they chewed off.

I got a little fanless atom PC from work that was being tossed because it "didn't work." It did work you would just have to apply pressure to it before boot because the CPU must have been coming unsoldered. I just stuck it on the bottom of like 30 pounds of gear and never had a problem.

Well at home i only have problems with spiders in my racks, not like the racks i have seen at work. Highlights are toiletpaper, chipmunks+nuts and worst in a school server rack a gun.

Mine would be when I had a 48 bay disk array / JBOD fail on me... badly. After a storm, it killed the larger card that allowed for me to get many of the drives into a PCIe 16x slot, and I was relegated down to only getting ~8 disks per box made from spare hardware. A single box I got 16 going. Add to this mix an SSD for mid-line caching.

These were all running bcache on top of mdraid... One single mount.

Yes I understand how obnoxiously stupid it was to run RAID6 on a 48 disk volume. It was almost all just stuff I could re-acquire over time, not irreplaceable things.

I just HAD to solve this one though.

In come several spare chassis / mobo etc.... get a bunch of drives powered and on /dev/ , move to the next.

A couple spare gigabit switches...

several gigabit NICs...

two explicit paths for each machine...

a bit of iSCSI magic, and one machine now had the physical disks all exposed to it... mdadm --assemble blah blah, bit of UUID chaos...

It's surprising that while a bit speed limited (I think I got just around 110MB/sec reads), it was nicely performant for what a huge mess of wires and disks just strewn out around my rack.

Managed to evacuate all I needed without much issue once I got that going. Now, I try to keep my arrays under 16 drives at a time, or keep a very rigid policy of "I can lose this and don't care" vs "this box gets RAID10 and/or offsite backups nightly".

back up your critical stuff people!

also mouse not pictured

😤😤😤