I commented elsewhere here, but E2E encryption is just between the server and the end user (ie a VPN)
You're thinking about encryption at rest, on the storage.
Immich would have to setup a whole new design to be able to store all the metadata on a per-user basis... but... you could have multiple Immich instances if you were to host it for your friends, but I think we're drifting into "why bother" now...