[–] 3 points 4 months ago* (last edited 4 months ago)

If an attacker breaks out of the container, through a kernel bug, a runtime bug, a careless bind mount, or a misconfigured capability, they can land on the host as root. This is not hypothetical, we have well known examples, such as CVE-2025-9074.

If the same happened with a rootless container, the attacker would be able to act in the host system as well, but with an unprivileged user, limiting the scope and impact of their “expedition”.

This is not true. I have addressed this in my other comment here: https://programming.dev/post/51095402/24088470

TLDR: Kernel bugs from inside a rootless container still get root on the host. Podman wasn't effected due to more restricted default seccomp policy (controlling what syscalls can be used), but podman being rootless didn't actually help in the example of that recent CVE.

  • source
  • [–] 2 points 4 months ago

    Built my first environment a couple of hours ago for a mission critical project. In this case, and a bit of overkill, we have hourly backups, and the process then spins up a VM, sets up pgsql, uploads the fresh backup, reports it as ok and then stores it, if it fails, requests a fresh backup, or calls the admin to report a problem.

  • source
  • [–] 2 points 4 months ago

    I noticed I could not restore pg dumps as expected when migrating to a new server. I had to add some parameter to pg dump so it's a better/more neutral dump or something, which was easier/able to restore. So - definitely check your backups.

    Trying the restore locally is also an option. Just to verify it works and the expected data is there.

    How important it is to verify and verify regularly depends on how much loss you're willing to accept.

    Once verified, as long as you don't do big version upgrades or automation changes, I would expect it to continue to work indefinitely. A semi regular verification still makes sense, though.

    At work I know my DB backups definitely work for other reasons, due to regular restores in non-prod.

    For side projects I definitely recommend verifying it at least once, when setting up automated backups/scripts for backup creation.

  • source
  • [–] 9 points 4 months ago*

    An untested backup is no backup at all. At least test once or twice per year.

    But I usually do it manually by restoring into a new instance, rather that have it done on a script.

  • source
  • [–] 2 points 5 months ago*

    Yeah, I think it's overkill unless you have 20+ nodes. When I switched from just a few VPSs with containers deployed with docker-compose, to deploying and administering a highly-available k8s cluster of 9 nodes, I'm pretty sure our cumulative uptime actually went down. There's ironically more points of failure that can bring stuff down due to the extra complexity.

  • source
  • [–] 6 points 5 months ago

    one user told The Register they had spending caps in place that should have stopped any bill over $250. Yet according to Google those caps can be automatically upgraded to $100,000 – without user input – if the user has spent a total of $1,000 throughout the life of the account, and the account is more than a month old.

    So a key way to protect themselves from this exact thing actually enables it. Something, something, cloud is cheaper?

  • source
  • [–] 3 points 7 months ago

    I was working on this for fb in 2020. We built a translation layer for each provider. All our infrastructure was in chef, it'd send the request to the translation layer, that would do witchcraft to give the providers the correct info.

  • source
  • [–] 2 points 10 months ago*

    https://home.robusta.dev/blog/stop-using-cpu-limits

    Okay, it's actually more complex than that. Because on self managed nodes, kubernetes is not the only thing that's running, so it can make sense to set limits for other non kubernetes workloads hosted on those nodes. And memory is a bit different from CPU. You will have to do some testing and YMMV but just keep the difference between requests and limits in mind.

    But my suggestion would be to try to see if you can get away with only setting requests, or with setting high very high limits. See: https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/#if-you-do-not-specify-a-memory-limit

    In order for them not to be OOM Killed, you have to set the memory requests for them above their highest spike, which means most of the time they’re only using like 25% or so of their memory allocation.

    Are you sure? Only limits should limit the total memory usage of a pod? Requests should happily let pods use more memory than the request size.

    One thing I am curious about is if your pods actually need that much memory. I have heard (horror) stories, where people had an application in Kubernetes with a memory leak, so what they did instead of fixing the memory leak, was to just regularly kill pods and restart new ones that weren't leaking yet. :/

    To answer your actual question about memory optimization, no. Even google still "wastes" memory by having requests and limits higher than what pods usually use. It is very difficult to prune and be ultra efficient. If an outage due to OOM costs more than paying for more resources would, then people just resort to the latter.

  • source
  • [–] 3 points 10 months ago

    Others are correct, the problem is the software. You are right to use memory requests and limits. The limits being the max it will use, but hopefully other pods won't be using all of their limits at once.

    So all of the pods' memory requests on a given node will sum to < 100% of the total available memory. So you can of course say your pod requests the highest amount of ram it will ever need, but that does mean it's reserved for that pod and won't be used anywhere else even during downtime

    K8s will allow over provisioning of ram for the limits though because it assumes it will not always need that as you are seeing.

    What you can do is to set a priority class on the pod so when it spikes and you don't have enough ram, it will kill some other pod instead of yours, but that makes other pods more volatile of course.

    There's many options at your disposal, you'll have to decide what works best for your use case.

  • source
  • [–] 4 points 10 months ago

    That seems like a problem of the application, no? If the workloads have memory leaks or are too eager to get memory to itself, then no cluster will be able to make it perform better.

  • source
  • view more: next ›