> I think that there are way worse dangers than "humanoid bombs"
Yes, I am much more concerned about the scope for ubiquitous surveillance and systematic domination that even fairly modest gains in AI will allow. Something along the lines of the Emergency society in A Deepness in the Sky.
This is too much. About a week ago I finished reading A Deepness in the Sky, and a couple of days after that I started reading Charles Stross' Accelerando. In that one I ran across this passage:
> "Cats," says Pamela. "He was hoping to trade their uploads to the Pentagon
> as a new smart bomb guidance system in lieu of income tax payments. Something
> about remapping enemy targets to look like mice or birds or something before
> before feeding it to their sensorium. The old kitten and laser pointer trick."
>
> Manfred stares at her, hard. "That's not very nice. Uploaded cats are a
> *bad* idea."
It's been forever since I read the book, but wasn't the whole point of the Emergency culture they didn't use AI and relied entirely on hyper-focused humans with enslavement implants?
If off-the-shelf AI has the intelligence of even a ten year old, imagine the amount of automated snooping that can go on.
Why worry about sifting for keywords in text messages when you can literally read all the text messages and infer meaning from them, even when it's deliberately obfuscated?
It could lead to a Brazil-like future where you're drawn in to a mess of trouble because you used too many euphemisms when texting your significant other. "I'm bringing home the package right now..."
What do we do to push back against this kind of thing? Drop off the grid as Stallman would have you try? Eject yourself from society as a whole? Or will it be practical at that point to have sufficiently private, well encrypted channels of communication that you won't have to worry too much about that sort of thing? Technology does cut both ways.
It doesn't require an emergent intelligence to cause a massive shift in the way we view technology. A number of low-level intelligences that can be easily replicated may be the first disruption.
