This vastly underestimates how big and interconnected modern LLM networks are. They are 100s of GBs in size, and transmit GBs of data between layers 100s of times per second. Having to transmit any part of the intermediate state of an LLM over a 1 Gb/s link would degrade the performance of an LLM by over 1000x. Current top-of-the-line datacenter GPUs are running about 4 TB/s total memory bandwidth, a limit LLMs are already pushing up against.

The kind of clustering you're talking about is being done: between multiple GPUs on the same machine. The slowdown of dropping to the measly 64 Gb/s speeds of the PCIe 5.0x16 bus is a huge performance hit and has to be done very carefully. The various parts of the computation are broken up and shuttled around.

It's the combination of massively-parallel compute (in the GPU cores) and insanely-fast, fully interconnected RAM that makes generative AI possible. The load characteristics aren't conducive to parallelization.