> Overtake and spread computation to security vulnerable computers (presumably, basically every computer)
You could backdoor computers, sure. Spread your own computation to them? You just can't get a better-than-GPT-4 model to run at real-time speeds decentralized over wide area networks. Literally impossible. There's not the bandwidth, not the local compute hardware, and no access to specialized inference hardware.
> Gain a physical presence by convincing humans to build critical physical components. For example by sending them emails and paying them for it.
Pay for it using what money?
> Use that presence to start a grey-goo like world takeover through replicating assemblers (they don't have to be tiny)
As someone who actually works on this, you have no idea what you are talking about.
1. Grey-goo scenarios are pure science fiction that were NEVER feasible, and known to be impossible even back in the 80's when the media misunderstood Drexler's work and ran with this half-baked idea. For a full treatment, see Drexler's own retrospective in his more recent book, Radical Abundance.
2. Nanotechnology is an extremely hard problem that is not in the slightest bit bottlenecked by compute power or intelligence capability. The things that are hard in achieving atomically precise manufacturing are not things that you can simulate on a classical computer (so a years-long R&D process is required to sort out), and there is no way to train an ML model to make better predictions without that empirical data.
People like Yudkowsky talk about AIs ordering genome sequences from bio labs and making first-generation nanotechnology by mixing chemicals in a test tube. This is pure fantasy and reflects badly on them as it shows how willing they are to generalize based on fictional evidence.
Yeah, this is Yud’s argument, but I just don’t get it. Does the technology to end the world by sending a couple emails around already exist? If so, why hasn’t the world ended?
People generally don't want to end the world. Those with the power to do so already are living generally good lives so they see little reason to potentially sacrifice the world for more power. AI could have completely different utility functions than people though, so an AI might have less qualms about ending the world.
A superintelligent AGI could easily follow this three step plan:
1. Optional: Overtake and spread computation to security vulnerable computers (presumably, basically every computer)
2. Gain a physical presence by convincing humans to build critical physical components. For example by sending them emails and paying them for it.
3. Use that presence to start a grey-goo like world takeover through replicating assemblers (they don't have to be tiny)
Now I'm not a superintelligent AGI, so there may be even simpler methods, but this already seems quite achievable and nearly unstoppable.