There's no option named variable refresh rate. But there is one called Adaptive sync. It's set to Always.
If VRR can't work in borderless windowed mode, I guess I'll have to give up on it for this game. Because I get distracting frame drops without Frame Gen. A bit unfortunate I'd have performance compromises with my brand new graphics card, but I heard this is normal for UE5 games.