Mine would be when I had a 48 bay disk array / JBOD fail on me... badly. After a storm, it killed the larger card that allowed for me to get many of the drives into a PCIe 16x slot, and I was relegated down to only getting ~8 disks per box made from spare hardware. A single box I got 16 going. Add to this mix an SSD for mid-line caching.

These were all running bcache on top of mdraid... One single mount.

Yes I understand how obnoxiously stupid it was to run RAID6 on a 48 disk volume. It was almost all just stuff I could re-acquire over time, not irreplaceable things.

I just HAD to solve this one though.

In come several spare chassis / mobo etc.... get a bunch of drives powered and on /dev/ , move to the next.

A couple spare gigabit switches...

several gigabit NICs...

two explicit paths for each machine...

a bit of iSCSI magic, and one machine now had the physical disks all exposed to it... mdadm --assemble blah blah, bit of UUID chaos...

It's surprising that while a bit speed limited (I think I got just around 110MB/sec reads), it was nicely performant for what a huge mess of wires and disks just strewn out around my rack.

Managed to evacuate all I needed without much issue once I got that going. Now, I try to keep my arrays under 16 drives at a time, or keep a very rigid policy of "I can lose this and don't care" vs "this box gets RAID10 and/or offsite backups nightly".

back up your critical stuff people!