> Will some kind of magic turn these AIs into independently acting agents once they get smart enough? I doubt it

This could just as easily be reworded circa 2016 or so to “will some kind of magic turn a large collection of matrix weights into something that seems to understand language at a human level after being exposed to enough of it?”

We don’t know squat about how intelligence or consciousness actually work, so those are effectively magic as well, yet they clearly exist in humans.