That sounds like a lack of port forwarding on at least one side. Ensure the vpn port is properly open on both sides. There is also an option you can add to the wireguard config for keepalive set it to something like 1min