AFAIK, there is currently no real equivalent for xrandr on any Wayland compositor.

Normally, you override the EDID rather than specifying modelines manually. Sway also has some limited custom resolution support, but it's just resolution and refresh rate, and not precise timings. Apparently the DRM backend supports modelines but that's not really Wayland by my understanding, more like Steam's gamescope mode.. Actually, the DRM backend is the default backend, so it should work. The other backends are used when sway is run inside another compositor.

I’ve also noticed that Grsync’s password prompt is broken on Wayland.

Forcing XWayland may help here. Otherwise, I assume the application is run as root which generally causes some issues with Wayland but potentially can be made to work.