That just defeats the IP part of the KVM and in that case you'd better stick with a traditional KVM.
Your setup depends entirely on your threat model. In my case in a normal state everything on the network is locked. The KVM is never used for normal ops, only rebooting and entering a disk encryption key in case I'm remote and have a failure. The KVM can only be accessed through a VPN. That limits my threat exposure to be well below my threat model. If I was Edward Snowden this might not be enough, but last I checked I'm not hunted by any state actor, rogue or not, so making sure the KVM is not accessible from the internet is enough.