The problem is, if someone does get physical access to the machine, you've just made breaking into it much easier.

Edit: that's assuming the smart card is using rfid.