Access patterns on neural nets are predictable long in advance. That means latency doesn’t matter much

While the bag of experts approach has allowed segmentation of memory into vram and dram, the access in the particular expert isn't predictable, access is spread across all the memory for that expert, and latency dramatically affects bandwith.