An issue with LLM inference is that the software architecture changes significantly, very quickly, and the combination of Nvidia’s software support + some architectural flexibility makes GPUs quite advantageous for that.

Just as an example, everyone saw KV cache as this existential limiting factor for LLMs. Now, it’s tiny. Another is the old dense model paradigm vs. the almost comically sparse ones coming out. Even matrix-multiply compute could be rendered irrelevant if Bitnet gains traction.

And that’s not to speak of training.