Perhaps proving the point here. That's not enough to eliminate the secret, the dangling commit will persist. Though this might be a nitpick, it's rather hard to get it from the remote without knowing the SHA.

> generate a new key

Is absolutely the right answer. If you pushed a key, you should treat it as already compromised and rotate it.

You also need to clear the caches of the remote

Yeah it doesn't seem hard to rewrite the commit history