> And when I write "I" you should read "anyone who has physical possession of this key".

Add a PIN or password to the key and now anyone who has physical possession of the key will need to know your secret in order to use it.

Yubikeys, for example, will wipe their credentials after 8 wrong password attempts.