IDK, a lot of that data center AI hardware is built to self-destruct after a few years. It's going to take time for the factories to retool before we see prices drop back down.

there will probably be a flood of cheap hardware/server equipment flooding the second hand market, yes

Unless you're planning on buying the whole rack you're not scooping up anything, except maybe the hard drives. The memory and GPUs are not consumer grade hardware. Some of the GPUs you technically could hook up to your PC, for example the Nvidia A6000, but its launch price was ~$4500 so even on the cheaper end you're shelling out probably $3000 for a card that (out the box) is marginally better than the 4070ti. Most of the GPUs and pretty much all of the memory is not compatible with your standard consumer grade MoBos.