Jesus that is bonkers. 480GB on one GPU? I get why (model size and context headroom)… but that’s gonna be closing in on $100k (MSRP of an NVDA B300 - iirc their current top-end - is $50-60k, and that only (hah) brings 288GB).

Tangentially: I really wish the big US LLM players would stop fucking brute forcing their development by throwing silicon at everything. It seems embedded/austere/efficient coding techniques and algorithms are just being straight up discarded and forgotten. Tbh, this is a pattern I’ve observed as a slow creep since the advent of AWS and The Cloud a couple decades ago - the “AI” bubble is just accelerating the trend. The narrative (irritatingly parroted by my manager) is always “make it work, then make it efficient later” - but it’s always put in the tech debt bucket, so “later” is actually “never”. China is absolutely running way ahead of the US these days in terms of nurturing incisive and efficient engineering culture - it’s not even close. The fact that it’s essentially borne out of ITAR restrictions on the most advanced GPU tech is incidental; really, in a pretty significant way, the US kinda unintentionally did them a favor by constraining their hardware tier, because it forced them to write better code to make up for it. And they have been writing better code for a while now, especially in the LLM domain.

They could have made 30 modern GPUs for that amount of ram. It's not about the money.

I wonder if those will support Vulkan for their eventual use in gaming when they become outdated