It's akin to nuclear weapons. If you do not develop them, then you'd be subject to the will of the ones that develops them first. So invariably you have to invest in AGI lest, an unsavory group develops it first.
Kind of, but the key difference between AGI and nuclear weapons is that we can control our nuclear weapons. The current state of AI safety is nowhere near the point where controlling an AGI is possible. More disturbingly, to me it seems likely that it will be easier to create an AGI than to discover how to control it safely.
>> The current state of AI safety is nowhere near the point where controlling an AGI is possible.
I just don't understand this logic though. Just.....switch it off. Unlike humans, computers have an extremely easy way to disable - just pull the plug. Even if your AGI is self-replicating, somehow(and you also somehow don't realize this long before it gets to that point) just....pull the plug.
Even Carmack says this isn't going to be an instant process - he expects to create an AGI with an intelligence of a small animal first, then something that has the intelligence of a toddler, then a small child, then maybe many many years down the line an actual human person, but it's far far away at this point.
I don't understand how you can look at the current or even predicted state of the technology that we have and say "we are nowhere near the point where controlling an AGI is possible". Like....just pull the plug.
I still don't understand. That presuposes that the AGI will just materialize out of thin air, and immediately have human level intelligence. That once day you just have a bunch of dumb computers, the next day you have an AGI hell-bent on escaping at all cost.
That's not going to happen - even Carmack believes so. The process to get AGI is going to take a long time, and we'll go through lots and lots of iterations of progressively more intelligent machines, starting with ones that are at toddler-level at best. And yes, toddlers are little monkeys when it comes to escaping, but they are not a global world ending threat.
They will inevitably reach the point of a world-ending threat, but I'm very confident Carmack is right that this won't happen so quickly we can't see the signs of danger.
There's a lot of policy we can do to slow things down once we get near that point, which is something rarely talked about among AI safety researchers, but the fundamental existential danger of this technology is obvious.
What if the AGI ran on a decentralized network that had a financial incentive to continue running? How would you "switch off" an AGI running on Ethereum? Especially when some subset of people will cry murder because the AGI seems like it might be sentient?
That seems like an extremely far fetched scenario to be honest. The comment I replied to sounds like the threat is immediate and real - your scenario does not sound like it.
>>How would you "switch off" an AGI running on Ethereum?
And where would the AGI get the funds to keep running itself on Ethereum?
>> Especially when some subset of people will cry murder because the AGI seems like it might be sentient?
Why is this a problem? People will and do cry murder over anything and everything. Unless there is going to be a lot of them(and there won't) - it's not an issue.
> And where would the AGI get the funds to keep running itself on Ethereum?
Well, perhaps Ethereum is too concrete. I meant, imagine that the AGI algorithm itself was something akin to the proof-of-work, and so individual nodes were incentivized to keep it running (as they are with BTC/ETH now). Then we wouldn't be able to just "switch it off", the same way we're not able to switch off these blockchain networks. And that's how SkyNet was born.
15 years ago I would have thought it was pretty far-fetched too. But seeing how the Bitcoin network can consume ~ever-increasing amounts of energy for ~no real economic purpose, and yet we can't just switch it off, has gotten me to think about how this could very well be the case for an AGI, and not that far in the future. It just has to be decentralized (and therefore transnational) and its mechanism bound up with economic incentive, and it will be just as unstoppable as Bitcoin.
I feel kind of bad even just planting this seed of a thought on the internet, to be honest :(
But how would it or it’s nodes even arrive on a consensus ? Suppose it was a self modifying code that took input from the world. How would you decide which node got to provide it input, and to verify that it ran the agi code faithfully ? How would forks be chosen to continue the chain ?
I would think AGI would first be born as a child of a human. Someone who thought they could train software to be more like them then their own biological children.
Or a new Bert-religion…
I mean there are lots of applications of ai. Probably the best is for future programmers to learn how to train it and use it.
deep learning is quite general and it’s ai. Consciousness and independence are irrelevant and not really needed.
On the off chance that you're serious: Even if you can pull the plug before it is too late, less moral people like Meta Mark will not unplug theirs. And as soon as it has access to the internet, it can copy itself. Good luck pulling the plug of the internet.
Worth noting that current models like Google LaMDA appear to already have access to the live Internet. The LaMDA paper says it was trained to request arbitrary URLs from the live Internet to get text snippets to use in its chat contexts. Then you have everyone else, like Adept https://www.adept.ai/post/introducing-adept (Forget anything about how secure 'boxes' will be - will there be boxes at all?)
I'm 100% serious. I literally don't understand your concern at all.
>>And as soon as it has access to the internet, it can copy itself.
So can viruses, including ones that can "intelligently" modify themselves to avoid detection, and yet this isn't a major problem. How is this any differenent?
>>Good luck pulling the plug of the internet.
I could reach down and pull my ethernet cable out but it would make posting this reply a bit difficult.
> So can viruses, including ones that can "intelligently" modify themselves to avoid detection, and yet this isn't a major problem. How is this any different?
I now regret spending half an hour writing a response to your earlier comment.
You can't tell why a mass deployment of motivated autonomous "NSO-9000s" might be more dangerous than a virus that just changes its signature to fool a executable scanner? I don't believe you.
If you honestly believe that an "intelligent" (as adaptive as a slime mold) virus is basically as dangerous as a maliciously deployed AGI then there is literally nothing an AGI could do that would make you consider safety is important during the endeavor of building one.
First of all, I'm sorry you regret replying to me, that's not something I aim for in HN discussions.
>>You can't tell why a mass deployment of motivated autonomous "NSO-9000s" might be more dangerous than a virus that just changes its signature to fool a executable scanner? I don't believe you.
So I just want to address that - of course I can tell the difference, but I just can't believe we will arrive at that level of crazy intelligence straight away. Like others have said - more like AGI with the capacity to learn like a small child, then years of training later they grow to be something like an adult. More human facsimile less HAL9000.
I only brought up self modifying viruses because that's the example of current tech that's extremely incentivised to multiply and avoid detection, that's its main reason for existence, and it does that very poorly.
> So can viruses, including ones that can "intelligently" modify themselves to avoid detection, and yet this isn't a major problem. How is this any differenent?
How long did it take to get Code Red or the Nimda worm off the internet? It's a different internet today, but it's also much easier to get access to a vast amount of computing power if you've got access to payments. Depending on the requirements to replicate, one could imagine a suddenly concious entity setting up cloud hosting of itself, possible through purloined payment accounts.
...and even if Mark's bot is dogshit, Musk will cheerfully sell embodied versions by the millions to replace workers, systematically handing over our industrial infrastructure first to an unrestrained capital class and then to the robots. I'm not even sure which will be worse.
It's midnight, so I'm not super keen on watching the whole thing(I'll get back to it this weekend) - but the first 7 minutes sounds like his argument is that if you build a humanoid robot with a stop button, the robot will fight you to prevent you pressing its own stop button if given an AGI? As if the very first instance of AGI is going to be humanoid robots that have physical means of preventing you from pressing their own stop button?
Let me get this straight - this is an actual, real, serious argument that they are making?
It's an (over)simplified example to illustrate the point (he admits as much near the end), if you want better examples it may be good to look up "corrigibility" with respect to AI alignment
But abstractly the assumptions are something like this:
* the AGI is an agent
* as an agent, the AGI has a (probably somewhat arbitrary) utility function that it is trying to maximize (probably implicitly)
* in most cases, for most utility functions, "being turned off" rates rather lowly (as it can no longer optimize the world)
* therefore, the AGI will try not to be turned off (whether through cooperation, deception, or physical force)
> I don't understand how you can look at the current or even predicted state of the technology that we have and say "we are nowhere near the point where controlling an AGI is possible". Like....just pull the plug.
We literally don't know how to stop effective optimizing processes, deployed in non-handcrafted environments, from discovering solutions and workarounds that satisfy the letter of our instructions but not the spirit. Even for "dumb" systems, we have to rely on noticing, then post-hoc disincentivizing, unwanted behaviors, because we don't know how to robustly specify objectives.
When you train a system to, for example, "stop saying racist stuff", without actually understanding what you're doing, all you get is a system that "stops saying racist stuff" when measured by the specific standard you've deployed.
Ask any security professional how seriously people take securing a system, let alone how ineffective they are at it. Now consider the same situation but worse because almost no one takes AI safety seriously.
If you nod solemnly at the words "safety" and "reliability" but don't think anything "really bad" can happen, you will be satisfied with a solution that "works on your machine". If you aren't deeply motivated to build a safe system from the start because you can always correct things late, you are not building a safe system.
It will be possible to produce economically viable autonomous agents without robustly specified objectives.
But hey, surely a smart enough system won't even need a robustly specified objective because it knows what we mean and will always want it just as much.
Surely dangerous behavior like "empowerment" isn't an instrumental goal that effective systems will simply rediscover.
Surely the economic incentives of automation won't encourage just training bad behavior out of sight and out of mind.
Surely in the face of overwhelming profit, corporations won't ignore warning signs of recurring dangerous behavior.
Surely the only people capable of building an AGI will always be those guaranteed to prioritize alignment rather than the appearance of it.
Surely you and every single person will be taking AI safety seriously, constantly watching out for strange behavior.
Surely pulling the plug is easy, whether AI runs on millions of unsecured unmonitored devices or across hundreds of money printing server farms.
Surely an AGI can only be dangerous if it explicitly decides to fool humans rather than earnestly pursuing underspecified objectives.
Surely it's easy to build an epistemically humble AGI because epistemic humility is natural or easy enough to specify.
Surely humanity can afford to delay figuring out how best to safely handle something so potentially impactful, because we always handle these things in time.
What makes humans special though? AI will be born on Earth so we will share that common trait with it, but perhaps one should have humility to accept that future doesn’t belong to a weak species like humans.
OTOH if you & your foes develop them both, then there is a probability asymptotically approaching 1 that the weapons will be used over the next X years. Perhaps the only winning move is indeed not to play?
