You could self-host a shared "source of truth" git repo that you access over ssh or filesystem. That can be anything from a USB thumb drive, a small clean server or a container on your existing desktop with ssh access, to an entire Forgejo deployment. Then you only need the "secret zero" of an ssh key to get everything set up and syncable.
If fresh setup is more common, you probably have other parts like package installation and network configuration that you also want to automate. Enter configuration management like ansible or salt, image builders like packer or archiso, "immutable" solutions like Nix or rpm-ostree. Once you get there you typically manage that in git anyway and you could put your dotfiles repo as a submodule and copy them over as part of OS setup.
If it's just for once in a blue moon, manual ad-hoc copying gets you pretty far.
No matter how you slice it I think you have to either frequently spend time syncing changes or just accept the drift and divergence between machines and the sources.