I don't have deep knowledge on the topic but you might want to look into EDID, which is how your system knows what resolutions the monitor supports. Incorrect detection of maximum resolutions and refresh rates is a semi-common problem on Linux and the fix for that is a custom modeline; setting one will be compositor dependent on Wayland (on X11 xrandr will do).
However it's also possible that all the cables you have are bad and/or don't support 8K.