I’m currently using Piraeus / LINSTOR and am quite happy with it: https://github.com/piraeusdatastore/piraeus
post
Openebs mayastor
But you could fit ceph on that I think. As long as your network between nodes is fast enough.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:
| Fewer Letters | More Letters |
|---|---|
| NFS | Network File System, a Unix-based file-sharing protocol known for performance and efficiency |
| SSD | Solid State Drive mass storage |
| k8s | Kubernetes container management package |
[Thread #95 for this comm, first seen 15th Feb 2026, 11:20] [FAQ] [Full list] [Contact] [Source code]
I’m not sure you’ll get nice performance in local network with small appliances (consumer network hardware, mini PCs and rpi 4). I’ve never got sub-ms network disk access on 1Gbps switch and router. In the end I’ve done the opposite - I’ve added one k8s host with a lot of storage, and any storage services are deployed there. All the other k8s services rely on local SSDs.
I have two storage nodes and one is much faster than the other.
I'm currently evaluating a juicefs deployment based on two minio instances (one per node, replicated with async bucket replication) through a load balancer (sidekick) in failover. Because juicefs also needs a db for metadata, I went with valkey + sentinel.
Juicefs provides a CSI driver that supports ReadWriteMany volumes and CSI snapshots and manages both read and write cache. Performance is much much better than Ceph. In theory it should be riskier (because of the async replication) but in practice I haven't yet lost a bit.
Thanks! A bit more involved that I’d have thought but still worth considering! Could you update us after your evaluation?
Well actually it is very easy to spin up in docker and most of the configuration happens through env variables.
juicefs itself only exists on the client side, so you basically only have to install and configure the CSI driver with helm.
as it took me a few days to come up with this solution I'd be happy to share my config files.
Performance wise is quite fast on sequential reads (it saturates my 2.5G bandwidth) and slower than I expected on sequential writes (for me it caps at 60MB/s). Postgresql seems happy. I saw no visible performance degradation with Authentic, Immich and Opencloud. Nextcloud installation took ages. I've yet to try it with jellyfish and the *arr suite.
A simple NFS share would be faster, but it doesn't support replication, failover and CSI snapshots.
Has anyone tried this https://github.com/awslabs/mountpoint-s3-csi-driver?
Perhaps with a Garage DaemonSet as a backend?
I'd expect the performance to be awful but it still has relatively niche usecases, especially where performance isn't a concern. I'm imagining legacy apps that don't speak S3.
all 20 comments