Containers run on "bare metal" in exactly the same way other processes on your system do. You can even see them in your process list FFS. They're just running in different cgroup's that limit access to resources.
Yes, I'll die on this hill.
Containers run on "bare metal" in exactly the same way other processes on your system do. You can even see them in your process list FFS. They're just running in different cgroup's that limit access to resources.
Yes, I'll die on this hill.
My NAS will stay on bare metal forever. Any complications there is something I really don’t want. Passthrough of drives/PCIe-devices works fine for most things, but I won’t use it for ZFS.
As for services, I really hate using Docker images with a burning passion. I’m not trusting anyone else to make sure the container images are secure - I want the security updates directly from my distribution’s repositories, and I want them fully automated, and I want that inside any containers. Having Nixos build and launch containers with systemd-nspawn solves some of it. The actual docker daemon isn’t getting anywhere near my systems, but I do have one or two OCI images running. Will probably migrate to small VMs per-service once I get new hardware up and running.
Additionally, I never found a source of container images I feel like I can trust long term. When I grab a package from Debian or RHEL, I know that package will keep working without any major changes to functionality or config until I upgrade to the next major. A container? How long will it get updates? How frequently? Will the config format or environment variables or mount points change? Will a threat actor assume control of the image? (Oh look, all the distros actually enforce GPG signatures in their repos!)
So, what keeps me on bare metal? Keeping my ZFS pools safe. And then just keeping away from the OCI ecosystem in general, the grass is far greener inside the normal package repositories.
A NAS as bare metal makes sense.
It can then correctly interact with the raw disks.
You could pass an entire HBA card through to a VM, but I feel like it should be horses for courses.
Let a storage device be a storage device, and let a hypervisor be a hypervisor.
I feel like this too. I do not feel comfortable using docker containers that I didn't make myself. And for many people, that defeats the purpose.
"What is stopping you from" <- this is a loaded question.
We've been hosting stuff long before docker existed. Docker isn't necessary. It is helpful sometimes, and even useful in some cases, but it is not a requirement.
I had no problems with dependencies, config, etc because I am familiar with just running stuff on servers across multiple OSs. I am used to the workflow. I am also used to docker and k8s, mind you - I've even worked at a company that made k8s controllers + operators, etc. I believe in the right tool for the right job, where "right" varies on a case-by-case basis.
tl;dr docker is not an absolute necessity and your phrasing makes it seem like it's the only way of self‐hosting you are comfy with. People are and have been comfy with a ton of other things for a long time.
Question is totally on purpose, so that you'll fill in what it means to you. The intention is to get responses from people who are not using containers, that is all. Thank you for responding!
Curious to know what the experiences are for those who are sticking to bare metal. Would like to better understand what keeps such admins from migrating to containers, Docker, Podman, Virtual Machines, etc. What keeps you on bare metal in 2025?
If it aint broke, don't fix it 🤷
I consider them unnecessary layers of abstraction. Why do I need to fiddle with Docker Compose to install Immich, Vaultwarden etc.? Wouldn't it be simpler if I could just run sudo apt install immich vaultwarden, just like I can do sudo apt install qbittorrent-nox today? I don't think there's anything that prohibits them from running on the same bare metal, actually I think they'd both run as well as in Docker (if not better because of lack of overhead)!
Both your examples actually include their own bloat to accomplish the same thing that Docker would. They both bundle the libraries they depend on as part of the build
Idk about Immich but Vaultwarden is just a Cargo project no? Cargo statically links crates by default but I think can be configured to do dynamic linking too. The Rust ecosystem seems to favour static linking in general just by convention.
Yes, that was my point, you (generally) link statically in Rust because that resolves dependency issues between the different applications you need to run. Cost is a slightly bigger, bloatier binary, but generally it's a very good tradeoff because a slightly bigger binary isn't an inconvenience these days.
Docker achieves the same for everything, including dynamically linked projects that default to using shared libraries which can have dependency nightmares, other binaries that are being called, etc. It doesn't virtualize an entire OS unless you're using it on MacOS or Windows, so the performance overhead is not as big as people seem to think (disk space overhead, though... can get slightly bigger). It's also great for dev environments because you can have different devs using whatever the fuck they prefer as their main OS and Docker will make everyone's environment the same.
I generally wouldn't put a Rust/Cargo project in docker by default since it's pretty rare to run into external dependency issues with those, but might still do it for the tooling (docker compose, mainly).
It's not just libraries in a docker
True, Docker does it better because any executables also have redundant copies. Running two different node applications on bare metal, they can still disagree about the node version, etc.
The actual old-school bloat-free way to do it is shared libraries of course. And that shit sucks.
I've been self-hosting since the '90s. I used to have an NT 3.51 server in my house. I had a dial in BBS that worked because of an extensive collection of .bat files that would echo AT commands to my COM ports to reset the modems between calls. I remember when we had to compile the slackware kernel from source to get peripherals to work.
But in this last year I took the time to seriously learn docker/podman, and now I'm never going back to running stuff directly on the host OS.
I love it because I can deploy instantly... Oftentimes in a single command line. Docker compose allows for quickly nuking and rebuilding, oftentimes saving your entire config to one or two files.
And if you need to slap in a traefik, or a postgres, or some other service into your group of containers, now it can be done in seconds completely abstracted from any kind of local dependencies. Even more useful, if you need to move them from one VPS to another, or upgrade/downgrade core hardware, it's now a process that takes minutes. Absolutely beautiful.
I started hosting stuff before containers were common, so I got used to doing it the old fashioned way and making sure everything played nice with each other.
Beyond that, it's mostly that I'm not very used to containers.
For me the learning curve of learning containers does not match the value proposition of what benefits they're supposed to provide.
I really thought the same thing. But it truly is super easy. At least just the containers like docker. Not kubernetes, that shit is hard to wrap your head around.
Plus if you screw up one service and mess everything up, you don’t have to rebuild your whole machine.
100% agree, my server has pretty much nothing except docker installed on it and every service I run is always in containers.
Setting up a new service is mostly 0% risk and apps can't bog down my main file system with random log files, configs, etc that feel impossible to completely remove.
I also know that if for any reason my server were to explode, all I would have to do is pull my compose files from the cloud and docker compose up everything and I am exactly where I left off at my last backup point.
Why would I want add overheard and complexity to my system when I don't need to? I can totally see legitimate use cases for docker, and work for purposes I use VMs constantly. I just don't see a benefit to doing so at home.
Main benefit of Docker for home is Docker compose IMO. Makes it so easy to reuse your configuration
Ok I’m arguing for containers/VMs and granted I do this for a living… I’m a systems architect so I build VMs and containers pretty much all the time time at work… but having just one sorta beefy box at home that I can run lots of different things is the way to go. Plus I like to tinker with things so when I screw something up, I can get back to a known state so much easier.
Just having all these things sandboxed makes it SO much easier.
I started self hosting in the days well before containers (early 2000's). Having been though that hell, I'm very happy to have containers.
I like to tinker with new things and with bare metal installs this has a way of adding cruft to servers and slowly causing the system to get into an unstable state. That's my own fault, but I'm a simple person who likes simple solutions. There are also the classic issues with dependency hell and just flat out incompatible software. While these issues have gotten much better over the years, isolating applications avoids this problem completely. It also makes OS and hardware upgrades less likely to break stuff.
These days, I run everything in containers. My wife and I play games like Valheim together and I have a Dockerfile template I use to build self-hosted serves in a container. The Dockerfile usually just requires a few tweaks for AppId, exposed ports and mount points for save data. That paired with a docker-compose.yaml (also built off a template) means I usually have a container up and running in fairly short order. The update process could probably be better, I currently just rebuild the image, but it gets the job done.
pff, you call using an operating system bare metal? I run my apps as unikernels on a grid of Elbrus chips I bought off a dockworker in Kamchatka.
and even that's overkill. I prefer synthesizing my web apps into VHDL and running them directly on FPGAs.
until my ASIC shuttle arrives from Taipei, naturally, then I bond them directly onto Ethernet sockets.
/uj not really but that'd be sick as hell.
Erm. I'd just say there's no benefit in adding layers just for the sake of it.
It's just different needs. Say I have a machine where I run a dedicated database on, I'd install it just like that because as said there's no advantage in making it more complicated.
I use Raspberry Pi 4 with 16GB SD-card. I simply don't have enough memory and CPU power for 15 separate database containers for every service which I want to use.
So, are you running 15 services on the Pi 4 without containers?
Have done it both ways. Will never go back to bare metal. Dependency hell forced multiple clean installs down to bootloader.
The only constant is change.
In my own experience, certain things should always be on their own dedicated machines.
My primary router/firewall is on bare metal for this very reason.
I do not want to worry about my home network being completely unusable by the rest of my family because I decided to tweak something on the server.
I could quite easily run OpnSense in a VM, and I do that, too. I run proxmox, and have OpnSense installed and configured to at least provide connectivity for most devices. (Long story short: I have several subnets in my home network, but my VM OpnSense setup does not, as I only had one extra interface on that equipment, so only devices on the primary network would work)
And tbh, that only exists because I did have a router die, and installed OpnSense into my proxmox server temporarily while awaiting new-to-me equipment.
I didn't see a point in removing it. So it's there, just not automatically started.
Same here. In particular I like small cheap hardware to act as appliances, and have several raspberry pi.
My example is home assistant. Deploying on its own hardware means an officially supported management layer, which makes my life easier. It is actually running containers but i don’t have to deal with that. It also needs to be always available so i use efficient “right sized” hardware and it works regardless whether im futzing with my “lab”
My example is home assistant. Deploying on its own hardware means an officially supported management layer, which makes my life easier.
If you're talking about backups and updates for addons and core, that works on VMs as well.
For my use case, I’m continually fiddling with my VM config. That’s my playground, not just the services hosted there. I want home assistant to always be available so it can’t be there.
I suppose I could have a “production “ vm server that I keep stable, separately from my “dev” vm server but that would be more effort. Maybe it’s simply that I don’t have many services I want to treat as production, so the physical hardware is the cheapest and easiest option
I've always done things bare metal since starting the selfhosting stuff before containers were common. I've recently switched to NixOS on my server, which also solves the dependency hell issue that containers are supposed to solve.
Every time I have tried it just introduces a layer of complexity I can't tolerate. I have struggled to learn everything required to run a simple Debian server. I don't care what anyone says, docker is not simpler or easier. Maybe it is when everything runs perfectly but they never do so you have to consider the eventual difficulty of troubleshooting. And that would be made all the more cumbersome if I do not yet understand the fundamentals of Linux system.
However I do keep a list of packages I want to use that are docker-only. So if one day I feel up to it I'll be ready to go.
Did you try compose scripts as opposed to docker run
I don't know. both? probably? I tried a couple of things here and there. it was plain that bringing in docker would add a layer of obfuscation to my system that I am not equipped to deal with. So I rinsed it from my mind.
If you think it's likely that I followed some "how to get started with docker" tutorial that had completely wrong information in it, that just demonstrates the point I am making.
Im a hobbiest who just learned how to self host my own static website on a spare laptop over the summer. I went with what I knew and was comfortable with which is a fresh install of linux and installing from the apt package manager.
As im getting more serious im starting to take another look at docker. Unforunately my OS package manager only has old outdated versions of docker I may need to reinstall with like ubuntu/debian LTS server something with more cutting edge software in repo. I don't care much for building from scratch and navigating dependency roulette.
What OS are you using?
Linux Mint 22
They can but - if their current setup meets their needs - why? There ain't nothing wrong with having a few simple spare laptops, each an isolated environment for a few simple home server tasks each.
Don't get me wrong - I too advocate for docker, particularly on new builds, or as a relatively turnkey solution to get started for novice friends, but the best setup is the one that works, and they sound like they got theirs where they want it.
All my services run on bare metal because its easy. And the backups work. It heavily simplifies the work and I don't have to worry about things like a virtual router, using more cpu just to keep the container...contained and running. Plus a VERY tiny system can run:
Without a single docker container. Its using around 10-20% of the RAM and doing a dd once in a while keeps everything as is. Its been 4 years-ish and has been working great. I used to over-complicate everything with docker + docker compose but I would have to keep up with the underlining changes ALL THE TIME. It sucked, and its not something I care about on my weekends.
I use docker, kub, etc...etc... all at work. And its great when you have the resources + coworkers that keep things up to date. But I just want to relax when I get home. And its not the end of the world if any of them go down.
Oh so the other 80% of your RAM can sit there and do nothing? My RAM is always around 80% or so as its caching stuff like it's supposed to.
Hahaha that's funny. I hope your not serous.
Unused RAM is wasted RAM
Welp OP did ask how we set it up. And for a family instance its good enough. The ram was extra that came with the comp. I have other things to do than optimize my family home server. There's no latency at all already.
It spikes when peertube videos are uploaded and transcoded + matrix sometimes. Have a good night!
Do you use any tools for management, such as Ansible or similar?
what do you run for RSS?
also, I hope you are not doing backups by dding an in use filesystem
Assuming you run Synapse, that uses more than 1.5GB RAM just idling, your system has at the very least 16GB of RAM... Hardly what I'd call "very tiny"
...ok so Im lying about my system for...some reason?
Synapse looks like its using 200M right now. It jumps to 1 GB when being heavily used, but I only use it for piefed and a couple of other local rooms. Honestly its not doing so much for us so we were thinking of getting rid of it. Its irritating to keep having to set up new devices and no one is really using it.
Peertube is much bigger running around 500MB just doing its thing.
Its a single family instance.
# ps -eo user,pid,ppid,cmd,pmem,rss --no-headers --sort=-rss | awk '{if ($2 ~ /^[0-9]+$/ && $6/1024 >= 1) {printf "PID: %s, PPID: %s, Memory consumed (RSS): %.2f MB, Command: ", $2, $3, $6/1024; for (i=4; i<=NF; i++) printf "%s ", $i; printf "\n"}}'
PID: 2231, PPID: 1, Memory consumed (RSS): 576.67 MB, Command: peertube 3.6 590508
PID: 2228, PPID: 1, Memory consumed (RSS): 378.87 MB, Command: /var/www/gotosocial/gotosoc 2.3 387964
PID: 2394, PPID: 1, Memory consumed (RSS): 189.16 MB, Command: /var/www/synapse/venv/bin/p 1.1 193704
PID: 678, PPID: 1, Memory consumed (RSS): 52.15 MB, Command: /var/www/synapse/livekit/li 0.3 53404
PID: 1917, PPID: 645, Memory consumed (RSS): 45.59 MB, Command: /var/www/fastapi/venv/bin/p 0.2 46680
It's so simple that it takes so much less time, one day I may move to Podman but I need to have the time to learn. I host Jellyfin
I run my NAS and Home Assistant on bare metal.
Both of those are much easier to manage on bare metal. Everything else runs virtualized on my Proxmox cluster, whether it's Docker stacks on a dedicated VM, an application that I want to run separately in an LXC, or something heavier in its own VM.
I'm curious why you feel these are easier to run on bare metal? I only ask as I've just built my first proxmox PC with the intent to run TrueNAS and Home Assistant OS as VMs, with 8x SAS enterprise drives on an HBA passed through to the TrueNAS VM.
Is it mostly about separation of concerns, or is there some other dragon awaiting me (aside from the power bills after I switch over)?
Anything I run on Proxmox, per my own requirements, needs to be hardware-agnostic. I have a 3-node cluster set up to be a "playground" of sorts, and I like being able to migrate VMs/LXCs between different nodes as I see fit (maintenance reasons or whatever).
Some services I want to run on their own hardware, like Home Assistant, because it offers more granular control. The Lenovo M710q Tiny that my HA system runs on, even with its i7-7700T, pulls a whopping 10W on average. I'll probably change it to the Pentium G4560T that's currently sitting on my desk, and repurpose the i7-7700T for another machine that could use the horsepower.
My NAS is where Im more concerned about separation of duties. I want my NAS to only be a NAS. OMV is pretty simple to manage, has a great dashboard, spits out SMART data, and also runs a weekly rsync backup command on my RAID to a separate 8TB backup drive. I'm currently in the process of building a "new" NAS inside a gutted HP server case from 2003 to replace the Mac mini/USB 4-bay drive enclosure. New NAS will have a proper HBA to handle drives.
or is there some other dragon awaiting me (aside from the power bills after I switch over)?
My entire homelab runs about 90-130W. It's pulled a total of ~482kWh since February (when I started monitoring it). That's 3x tiny/mini/micro PCs (HP 800 G3 i7, HP 800 G4 i7, Lenovo M710q i7), an SFF (Optiplex 7050 i7), 2014 Mac mini (i5)/loaded 4-bay HDD enclosure/8TB USB HDD, Raspberry Pi 0W, and an 8-port switch.
Wow, thanks so much for the detailed rundown of your setup, I really appreciate it! That's given me a lot to think about.
One area that took me by surprise a little bit with the HBA/SAS drive approach I've taken (and it sounds like you're considering) is the power draw. I just built my new server PC (i5-8500T, 64GB RAM, Adaptec HBA + 8x 6TB 12GB SAS drives) and initial tests show on its own it idles at ~150W.
I'm fairly sure most of that is the HBA and drives, though I need to do a little more testing. That's higher than I was expecting, especially since my entire previous setup (Synology 4-bay NAS + 4x SATA drives, external 8TB drive, Raspberry Pi, switch, Mikrotik router, UPS) idles at around 80W!
I'm wondering if it may have been overkill going for the SAS drives, and a proxmox cluster of lower spec machines might have been more efficient.
Food for thought anyway... I can tell this will be a setup I'm constantly tinkering with.
In my case it’s performance and sheer RAM need.
GLM 4.5 needs like 112GB RAM and absolutely every megabyte of VRAM from the GPU, at least without the quantization getting too compressed to use. I’m already swapping a tiny bit and simply cannot afford the overhead.
I think containers may slow down CPU<->GPU transfers slightly, but don’t quote me on that.
Can anyone confirm if containers would actually impact CPU to GPU transfers
To be clear, VMs absolutely have overhead but Docker/Podman is the question. It might be negligible.
And this is a particularly weird scenario (since prompt processing literally has to shuffle ~112GB over the PCIe bus for each batch). Most GPGPU apps aren’t so sensitive to transfer speed/latency.
I generally abstract to docker anything I don't want to bother with and just have it work.
If I'm working on something that requires lots of back and forth syncing between host and container, I'll run that on bare metal and have it talk to things in docker.
Ie: working on an app or a website or something in language of choice on framework of choice, but postgres and redis are living in docker. Just the app I'm messing with and it's direct dependencies run outside.
TrueNAS is on bare metal has I have a dedicated NAS machine that's not doing everything else and also is not recommended to virtualize. Not sure if that counts.
Same for the firewall (opnsense) since it is it's own machine.
Anything you want dedicated performance on or require fine tuning for a specific performance use cases. Theyre out there.
I would always run proxmox to set up docker VMs.
I found Talos Linux, which is a dedicated distro for kubernetes. Which aligned with my desire to learn k8s.
It was great. I ran it as bare-metal on a 3 node cluster. I learned a lot, I got my project complete, everything went fine.
I will use Talos Linux again.
However next time, I'm running proxmox with 2 VMs per node - 3 talos control VMs and 3 talos worker VMs.
I imagine running 6 servers with Talos is the way to go. Running them hyperconverged was a massive pain. Separating control plane and data/worker plane (or whatever it is) makes sense - it's the way k8s is designed.
It wasn't the hardware that had issues, but various workloads. And being able to restart or wipe a control node or a worker node would've made things so much easier.
Also, why wouldn't I run proxmox?
Overhead is minimal, get nice overview, get a nice UI, and I get snapshots and backups
What are you doing running your vms on bare metal? Time is a flat circle.
There’s one thing I’m hosting on bare metal, a WebDAV server. I’m running it on the host because it uses PAM for authentication, and that doesn’t work in a container.
I have a single micro itx htpc/media server/nas in my bedroom. Why use containers?
@kiol I mean, I use both. If something has a Debian package and is well-maintained, I'll happily use that. For example, prosody is packaged nicely, there's no need for a container there. I also don't want to upgrade to the latest version all the time. Or Dovecot, which just had a nasty cache bug in the latest version that allows people to view other peoples' mailboxes. Since I'm still on Debian 12 on my mail server, I remain unaffected and I can let the bugs be shaken out before I upgrade.
@kiol On the other hand, for doing builds (debian packages and random other stuff), I'll use podman containers. I've got a self-built build environment that I trust (debootstrap'd), and it's pretty simple to create a new build env container for some package, and wipe it when it gets too messy over time and create a new one. And for building larger packages I've got ccache, which doesn't get wiped by each different build; I've got multiple chromium build containers w/ ccache, llvm build env, etc
@kiol And then there's the stuff that's not packaged in Debian, like navidrome. I use a container for that for simplicity, and because if it breaks it's not a big deal - temporary downtime of email is bad, temporary downtime of my streaming flac server means I just re-listen to the stuff that my subsonic clients have cached locally.
@kiol Syncthing? Restic? All packaged nicely in Debian, no need for containers. I do use Ansible (rather than backups) for ensuring if a drive dies, I can reproduce the configuration. That's still very much a work-in-progress though, as there's stuff I set up before I started using Ansible...
top 99 comments