That's not sufficient. If you can access the device, then you can access the remotely managed server. Since it's remotely managed it must be an important server and the KVM can bypass traditional security measures since the KVM gives the same level of access as physical access. If you get through to the server, then isolating the KVM doesn't matter.
What I do is I treat physical access as inherently insecure as if anyone could drop by. Encryption keys still need to be presented and the only screen shown when there's no activity is a login screen. The KVM is treated like any other random access to the server, physical or SSH: You must identify yourself.