I guess the most efficient option would be to have a distributed setup where most households have some local compute, which they make available for others to use when they don't need it themselves. Kinda like with SETI@home or Folding@home.

There are 2 type: Distributed (like https://en.wikipedia.org/wiki/Folding@home )

And centralized (like every single data center on Earth)

The first lets people prioritize what they want and the second is more efficient in terms of resources.

The old petals framework did that via a torrent like protocol and could even train models over a distributed network, but that was years ago and I have no clue if such a system is viable with all the new features in new models.

My guess is that we could exploit a distributed kv-cache ala deepseek, but they use their own proprietary algorithm for kv storage/retrieval and a huge datacenter, and with only normal internet connections, I'm not sure its viable fir people ?

Big models are a problem, but something like aihorde with smaller models combined with a distributed agentic harness could be interesting.