Hacker reverse engineers his Samsung TV and finds on-device AI listening for your dog and baby, watching what you watch, and borrowing your soundbar's mic if the TV doesn't have one. Then OpenAI banned him for reverse-engineering it.
Source:
-
Exl3 quantization at 3.4bpw or so, KLD very close to the higher Q3/close to Q4 GGUFs.
-
TabbyAPI as a backend
-
Split each layer, experts are profiled and hot ones are run on GPU during hybrid inference.
I have a Ryzen 7800 with 128GB DDR5, which is how it fits. The “cold” MoE experts are run on the CPU, but this is very sparse, which is how it stays fast. Batch size of 8192 keeps prompt processing fast even though it is loading experts from RAM for that.
There’s also a lot of system optimization to get it there, like allocating large pages at boot, running the backend at a high priority, keeping background RAM/VRAM usage minimal so it will fit, sizing the quantization just right and such.
And this was not possible… a few weeks ago? DeepseekV4 just came out, and exllama got support for hybrid inference very recently. Before that, I could only get 8-12 toks/sec with a big MoE like this.