Just to reiterate the parent: This is only valuable if we trust the signatures - which I wouldn't if they were, say, just held along side the "hacked" gems server.
You still need the public key to validate the signature. If the attacker can change the public key, he can change the signature without you knowing - unless you explicitly want to trust each and every key for every gem you install.