Yeah, the connector spec is definitely bad given the incidents we've seen. I was just pointing out it's not actually the mating cycle limit that's the issue here (especially given older connectors like the 8 pin evidently don't have this problem, despite equivalent mating cycle ratings), but rather the rest of the spec.

FWIW, lots of power connectors are indeed only rated for a low number of mating cycles (only 30), not just 12VHPWR. This includes the old PCIe 8 pin (same 30 limit). Realistically you will probably not reach the rated limit.

The actual problem, I think, is that the connector could operate without being fully plugged in, especially without current balancing on the side of the card. This is an actual fire hazard, as you now get at least some connections with very high resistance, forcing extreme current on the remaining wires with low resistance connections. This is combined with the fact that the safety margin on 12VHPWR is a lot lower than that of the PCIe 8 pin (which I believe can safely carry a lot more than its 150W rating).

It still won't help for a couple of reasons. For one, VRAM is fairly abundant on commercial deployments. Even for big models, a company is probably deploying on 1-2 nodes of 8x H200 or newer (hence several TB of VRAM).

But more importantly, commercial inference relies on heavy concurrency. So even if some experts are uncommon, with a lot of concurrent users, they will still fire frequently enough for the performance difference to be felt. And in my own experience, expert use isn't uniform but it isn't particularly biased either. This is especially tough since high-concurrency inference can actually be fairly compute bound, but this expert caching system either starves the system of bandwidth (if you require compute on the GPU, then you're stuck with PCIe speeds which are tiny compared to HBM and even DRAM), or you're starved of compute (if you do compute on the CPU).

It's nice for local inference, but yeah, not representative of commercial inference.

Keep in mind many of the optimizations you're talking about (LRU caching of experts, for example) are only really relevant at the single user local inference scale. As in, an individual wants to run a big model on their machine, but they don't have enough VRAM to fit the model and KV cache. Accordingly, you're basically only describing hobbyist and research projects, which aren't really representative of the AI inference industry as a whole.

Commercial inference keeps everything resident in VRAM, so expert caching isn't necessary. So these things won't help costs. A lot of other low hanging fruit (like hierarchical KV cache) has also existed for a long time for production-ready inference engines.

Yeah I can believe their interconnect is better, given their extensive history in networking.

W.r.t TFLOPs, let me clarify what I meant. Even on traditionally compute-bound workloads (attention, etc.), on H200 it's actually surprisingly difficult to make full use of the card's throughput before hitting VRAM bandwidth limits. Tensor core throughput has grown a lot faster than bandwidth has.

I've never written a kernel for Huawei chips so I have no idea if they have the same problem. But this problem is there on many datacenter-class NVIDIA chips, which is why they keep introducing features (TMA, TMEM, etc.) to try and lower the time wasted waiting for memory.

You can actually get kind of acceptable performance on CPU alone, but you need rather specific CPUs, like SPR or newer Intel Xeons. These support AMX, which is almost like a mini tensor core, so you can actually get decent throughput in TFLOPs out of GNR Xeons. Memory bandwidth with max channels is also acceptable, something like ~800 GB/s per socket with maxed out MRDIMMs, which is not too far behind consumer GPUs like 3090 and 4090.

Not anywhere near the performance of real GPUs of course, and not something acceptable for scale or production workloads, but good enough for local inference.

Makes sense, even Flash is fairly sizable! KTransformers also has a "llamafile" backend which uses GGUFs, but ik_llama will almost certainly perform better if you're not on a NUMA setup. In my case, I'm using a dual socket motherboard, so KTransformers performs quite a bit better (I think ik_llama hasn't implemented extensive NUMA optimizations quite yet, but sounds like it's coming), though I normally use KTransformers for native FP8 weights.

Yeah, I'd expect KTransformers to add support eventually, especially considering their existing support for previous DeepSeek models. One of the tricky parts is that backends need both FP8 and MXFP4 support. As far as I'm aware no inference engine supports both on CPU at the moment (llama.cpp added fp4 support recently, but doesn't have fp8, while kt-kernel doesn't support fp4 yet).

To be fair, the raw FLOPs count doesn't tell the whole story. On a lot of workloads (including token generation during LLM inference), you're bound by the memory bandwidth rather than throughput/FLOPs. On H100/H200, keeping the tensor cores fully occupied is surprisingly difficult, and that's with 3+ TB/s of memory bandwidth. And I believe those cards have much higher throughput (at least at FP8, Ascend wins at FP4 since H100/200 don't support it) compared to Ascend.

The Ascend 950PR units have far lower memory bandwidth, reportedly at 1.4 TB/s. Compare that to Blackwell, which has something like 8TB/s of bandwidth. I believe they're manufacturing their own kind of HBM, so that's still really impressive considering this is a fairly recent push into manufacturing accelerators. But I'm a bit skeptical it actually outperforms NVIDIA at scale.

What info have you heard about Fenghua 3? I'd last read that it's not strictly an AI accelerator but can actually do graphics tasks, which is neat. Would make it more of a competitor to a professional workstation card like an RTX PRO 6000.

I'm most curious about their CUDA compatibility claim. I would expect that to cause a pretty significant performance hit since when writing high-performance CUDA kernels, you generally need to specialize the kernel to the individual GPU (an H100 kernel will look quite different compared to a 4090 kernel, for example). But if in spite of that it can achieve H100 performance, that'd be cool.

Like others have mentioned, the spider (the wires) and the secondary do shadow some light that would otherwise reach the primary. It also results in some artifacts due to diffraction; the view ends up convolved with the Fourier transform of the aperture. This is why on Hubble images, you see cross shaped stars, as that's the shape of the Fourier transform of its 4-strut spider.

Every time I see a headline like this I’m reminded of the time I heard someone describe the modern state of AI research as equivalent to the practice of alchemy.

Not sure if you're referencing the same thing, but this actually came from a presentation at NeurIPS 2017 (the largest and most prestigious machine learning/AI conference) for the "Test of Time Award." The presentation is available here for anyone interested. It's a good watch. The presenter/awardee, Ali Rahimi, talks about how over time, rigor and fundamental knowledge in the field of machine learning has taken a backseat compared to empirical work that we continue to build upon, yet don't fully understand.

Some of that sentiment is definitely still true today, and unfortunately, understanding the fundamentals is only going to get harder as empirical methods get more complex. It's much easier to iterate on empirical things by just throwing more compute at a problem than it is to analyze something mathematically.