I think gwern's point (whose opinion on deep learning I respect a lot more than his on genetics) is that GPT-3 is nearly not the end of the story. OpenAI released a GPT pretty much every year and each one is a spectacular improvement on the previous one with no sign of that trend ever stopping. If size is really what matters at all, there's no telling what GPT-4 or GPT-5 might be capable of, let alone a giant GPT run by state-sized actors.
I think the point Gwern makes is that if a government wanted to they could easily allocate enough resources to do this. More bluntly I think he's saying if the US decided tomorrow to begin a Manhattan Project for AGI there is a non-zero chance that they might succeed in 7 years.
Lots of large companies don't include their clusters on those supercomputer lists. Including Google. IDK about DoD, but I wouldn't be surprised if they have top 10 clusters we don't know about. The list of largest supercomputers is definitely only a list of the largest disclosed supercomputers.
That said, only so many actors could've secretly sourced 10K+ v100s with fast interconnect...
