There were plenty of open weights LLMs, image, audio, or video generators that you can run locally on a decent NVIDIA GPU, if that's what you mean. Try the !localllama@sh.itjust.works community

Already done. See:

Open training: https://huggingface.co/collections/allenai/olmo-3

Decentralized training: https://huggingface.co/NousResearch/Hermes-4.3-36B

Decentralized inference: https://aihorde.net/

They aren’t the only example of each, either.

The issue, as with many decentralized projects, is that they're isolated from each other, and too few know about them.

If you mean distributing inference across many machines, each of which could not individually deal with a large model, using today's models, not viable with reasonable performance. The problem is that you require a lot of bandwidth between layers; a lot of data moves. When you cluster current systems, you tend to use specialized, high-bandwidth links.

It might theoretically be possible to build models that are more-amenable to this sort of thing, that have small parts of a model run on nodes that have little data interchange between them. But until they're built, hard to say.

I'd also be a little leery of how energy-efficient such a thing is, especially if you want to use CPUs

which are probably more-amenable to be run in a shared fashion than GPUs. Just using CPU time "in the background" also probably won't work as well as with a system running other tasks, because the limiting factor isn't heavy crunching on a small amount of data

where a processor can make use of idle cores without much impact to other tasks

but bandwidth to the memory, which is gonna be a bottleneck for the whole system. Also, some fairly substantial memory demands, unless you can also get model size way down.

Have you heard about the AIhorde project by the DBzero community ?

@aihorde@lemmy.dbzer0.com draw for me a surprised lemming hacker typing in front of a green text computer screen

It's a capitalist world. Economies of scale will always tip the balance toward centralization, in the hands of billionaires.

Plausibly feasible, but there would be more people using the service than letting the service borrow their hardware for processing or memory. I imagine it would work like a botnet, where if a user generates a prompt, their machine would borrow other nearby machines to process the command efficiently. So, in order to make it a fair service, perhaps the software does a POST and only operates if you meet a minimum hardware specification, and a stable internet connection.