I actually made a similar thing a few years ago, and mostly relied on WebSockets with a HTTP fallback: https://github.com/Steve-Tech/My_Time

You actually seem to know what you're doing with Workers though, mine was pretty much a half-assed attempt at porting my python code (which again could probably be better).

Also my desktop is PTP synced to a GNSS disciplined time server, and I've found the My_Time workers demo to be fairly accurate. I also feel Cloudflare should be reasonably accurate given they have database products you're intended to use with workers, but yeah, there's no guarantees.