Let's tinker around and accidentally break something.
post
replies:
If itβs stable, itβs not a lab.
Thatβs infrastructure.
replies:
When's the last time you checked if your backup solution works?
replies:
logging is probably down
You do, of course have a dedicated rsyslogd server? An isolated system to which logs are sent, so that if someone compromises another one of your systems, they can't wipe traces of that compromise from those systems?
Oh. You don't. Well, that's okay. Not every lab can be complete. That Raspberry Pi over there in the corner isn't actually doing anything, but it's probably happy where it is. You know, being off, not doing anything.
replies:
You have remote power management set up for the systems in your homelab, right? A server set up that you can reach to power-cycle other servers, so that if they wedge in some unusable state and you can't be physically there, you can still reboot them? A managed/smart PDU or something like that? Something like one of these guys?
Oh. You don't. Well, that's probably okay. I mean, nothing will probably go wrong and render a device in need of being forcibly rebooted when you're physically away from home.
replies:
Time to start documenting it!
replies:
Have you already tried implementing an identity provider like Authentik, so you can add OIDC and ldap for all your services, while you are the only one thatβs using them? π€
replies:
All of those systems in your homelab...they aren't all pulling down their updates multiple times over your network link, right? You're making use of a network-wide cache? For Debian-family systems, something like Apt-Cacher NG?
Oh. You're not. Well, that's probably okay. I mean, not everyone can have their environment optimized to minimize network traffic.
replies:
The rare moment when everything actually works. π
replies:
Don't worry, you're one Docker pull away from having to look up how to manually migrate Postgres databases within running containers!
(Looks at my PaperlessNGX container still down. Still irritated.)
replies:
I should do some breaking network changes... While tunneled in.
replies:
Backups. You're forgetting them.
replies:
Now try migrating all your docker containers to podman.
replies:
You have an intrusion detection system set up, right? A server watching your network's traffic, looking for signs that systems on your network have been compromised, and to warn you? Snort or something like that?
Oh. You don't. Well, that's probably okay. I mean, probably nothing on your network has been compromised. And probably nothing in the future will be.
Never run:
docker compose pull
docker compose down
docker compose up -d
Right before the end of your day. Ask me how I know π
replies:
All of your systems are set up, but are they capable of being redeployed using a configuration management software package? Ansible or something like that?
Oh. They're not. Well, that's probably okay. I mean, you could probably go manually reproduce configurations, more or less.
You have all your devices attached to a console server with a serial port console set up on the serial port, and if they support accessing the BIOS via a serial console, that enabled so that you can access that remotely, right? Either a dedicated hardware console server, or some server on your network with a multiport serial card or a USB to multiport serial adapter or something like that, right? So that if networking fails on one of those other devices, you can fire up minicom or similar on the serial console server and get into the device and fix whatever's broken?
Oh, you don't. Well, that's probably okay. I mean, you probably won't lose networking on those devices.
replies:
view more:
next βΊ
top 96 comments