Sure, security adds friction. Whether the added friction is worth the added security in your case depends on the amount of friction and the amount of security it buys you. The trade off is likely different for you than it is for a larger org. For example, you (or the admins responsible for your org) could maintain an apt repo with signed nvidia drivers for your org. This would reduce the friction by centralizing the signing process.

I keep my signing key on my machine, but gpg-encrypted bound to a yubikey. Is that frictionless? No, certainly not. Does it provide perfect security? No, certainly not. A dedicated attacker can root my box and wait until I need to sign a module. Does it protect me from loading random kernel modules if I get hit by an automated attack? Most likely. Good enough for what I currently expect as threats.