There are efforts there. The new Deepseek 4 compresses a lot of its knowledge using something they call engrams. But it's unfortunately still too big for a consumer GPU.

Gemma 4 is small enough to run on your cellphone.

If your GPU has at least 8GB there are a lot of options for self hosting your own local models