You can extend the matrix across so it becomes one board, but it means custom wiring the connector.
Yeah, physically that will work. I should have mentioned that, as it's what I did (permanently wired) on my failure of an ergo split to get it split at all with one MCU, but electrically it worked perfectly fine. Proper USB-C/Thunderbolt like the caravan 2 seems like it would be a good solution for a PCB, and I'm curious what issues the mini HDMI produced (crosstalk maybe, or just complaining about sourcing mini to mini cables?). Ethernet or DB9 should do for hand-wired, though obviously bulkier.

