Keys are not the only real security... real security includes something you are, something you have, and something you know.
Having something like this combined with the something you know (passphrase) would be real closer security. Now anyone with your token can access everything that key gets access to, without you needing to be there, or sharing a passphrase. That's less secure imho.
A lot of these devices allow you to configure a pin code that is required to unlock the device before use, which effectively provides a second factor. Also, stealing a fido device requires physical theft and can't be duplicated, so the owner would likely know if it was stolen.
As xur17 points out, these devices support an on-device PIN like smart cards. The protocol also has support for future devices with biometric authentication, which could give you all three factors in one device.
Having something like this combined with the something you know (passphrase) would be real closer security. Now anyone with your token can access everything that key gets access to, without you needing to be there, or sharing a passphrase. That's less secure imho.