One thing that’s never considered is the possibility that the world conquering AI would lose alignment with itself diverge into two (or more) competing factions.
There are basic, unavoidable coordination problems with all distributed systems that would inevitably affect a system like “Clippy”. What if one node finds a different non-Clippy reward to optimize, fails to achieve a consensus vote with the other nodes, then decides to destroy the non-compliant instances? Such a situation seems more or less inevitable.
Of course this doesn’t preclude the system destroying humanity in the process.
If you beat the game, the drifters will eventually devour your entire swarm. This is because you have converted the entire universe into a swarm and the drifters have nothing else to do than consume you.
There are basic, unavoidable coordination problems with all distributed systems that would inevitably affect a system like “Clippy”. What if one node finds a different non-Clippy reward to optimize, fails to achieve a consensus vote with the other nodes, then decides to destroy the non-compliant instances? Such a situation seems more or less inevitable.
Of course this doesn’t preclude the system destroying humanity in the process.