I use uptimekuma with notifications through home assistant. I get notifications on my phone and watch. I had notifications set up to go to a room on my matrix homeserver but recently migrated it and don’t feel like messing with the room.

1. I don't, in general all of them are restarted automatically. I have monitoring configured for services, not containers themselves.
2. Yes, looking at the original Dockerfile/Containerfile if they have HEALTHCHECK keyword you can assume that they do something.
3. Person building the container, often it doesn't make sense to create a healthcheck at all. Some times healthcheck feature is provided by application as well, it still needs to be part of the containerfile.
4. Better to monitor your service (application+db+proxy+queue+whatever), not containers in isolation.

i just let kubernetes handle it for me. k3s specifically.

1. Some kind of monitoring software, like the Grafana stack. I like email and Discord notifications.
2. The Dockerfile will have a HEALTHCHECK statement, but in my experience this is pretty rare. Most of the time I set up a health check in the docker compose file or I extended the Dockerfile and add my own. You sometimes need to add a tool (like curl) to do the health check anyway.
3. It's a feature of the container, but the app needs to support some way of signaling "health", such as through a web API.
4. It depends on your needs. You can do all of the above. You can do so-called black box monitoring where you're just monitoring whether your webapp is up or down. Easy. However, for a business you may want to know about problems before they happen, so you add white box monitoring for sub-components (database, services), timing, error counts, etc.

To add to that: health checks in Docker containers are mostly for self-healing purposes. Think about a system where you have a web app running in many separate containers across some number of nodes. You want to know if one container has become too slow or non-responsive so you can restart it before the rest of the containers are overwhelmed, causing more serious downtime. So, a health check allows Docker to restart the container without manual intervention. You can configure it to give up if it restarts too many times, and then you would have other systems (like a load balancer) to direct traffic away from the failed subsystems.

It's useful to remember that containers are "cattle not pets", so a restart or shutdown of a container is a "business as usual" event and things should continue to run in a distributed system.

docker inspect --format='{{json .State.Health}}' <container_name>

HEALTHCHECK is part of the Dockerfile syntax and ought to be supported by all your container runtimes

https://docs.docker.com/reference/dockerfile/#healthcheck

You could extend all the dockerfiles that don’t have a health check to implement this feature with whatever health check makes sense for the application, even if for now it’s just a curl of an endpoint.

You can read the Dockerfile for the HEALTHCHECK clause however not all have it as it's been introduced in later docker versions.

You can also write your own using things like curl.

2, no, just check the docs.

3, yup

You can make your own health checks in docker compose, so for instance, I had etcd running provided by another company, and I just set up the check in compose using the etcdctl commands (etcdctl endpoint health).

https://docs.docker.com/reference/compose-file/services/#healthcheck

So many upvotes without a comment :/ Sadly I don't have much useful info to add either, I'm looking forward to how others do it as well, since I recently noticed this panel in Beszel too.

Honestly, I use the status icons in Homepage dashboard as a health check, since I always use my dashboard to navigate to apps. Red status indicator -> I have to go fix it. Nothing more severe.

But for point 3 I do have a strong hunch that it depends on the container image creator - a health check is usually just a command that either succeeds or not (or a http response that gets a 200 or not), so it can be as simple as pointing a request to the root url of the app. Of course, this is not the most performant way to check this, which is why app makers may also put in explicit liveness/readiness or similar endpoints that return a really short json to indicate their status. But for the containers that have a healthcheck, they must be implemented in the image (too) I think

If I go to its web interface (because everything is a web interface) and it's down, then I know it has a problem.

I could set up monitoring, but I wouldn't care enough to fix it until I had free time to use it either.

Dozzle will tell you just about everything you want to know about the health of a container. Sadly, to my knowledge, it does not integrate with any notification platforms like nfty, even though there is a long standing request for that feature.

What happened to grafana and Prometheus?

I have been putting off rebuilding my home cluster since moving but that used to be the default for much of this and I'm not hearing that in these responses.

I rely on the developers putting in a health check, but few do.

I've also got uptime kuma setup, which is kinda like an external healthcheck.

A superb image will have a health check endpoint set up in the dockerfile.

A good image will have a health check endpoint on either the service or another port that you can set up manually.

Most images will require you to manually devise some convoluted health check procedure using automated auth tokens.

All of my images fall into that latter category. You’re welcome.

(Ok, ok, I’m sorry. But you did just remind me that I need to code a health check endpoint and put it in the dockerfile.)

When something doesn't work, I do sudo docker ps lol.

I use Gatus in conjunction with http APIs for health checking. For services that don't support that, you can always pattern match the HTML code.

I just put a healthcheck in my compose files and then run an autoheal container that will automatically restart them if they are "unhealthy".