Such a system seems like it would be incredibly fragile to local attack - and this is one case where you can't just assume "physical access means you've already lost".
I agree, thats why I figured if you can get away with fooling around with a lock, some wires and a laptop in the hallway, you can probably pick the backup key more discreetly.
<Grant access to KEY_ID>
<Revoke access from KEY_ID>
And it would keep track internally so that if the central system went down it could still function with already issued keys until it is fixed.