I actually read this paper a couple weeks ago as part of a deep learning reading group that I co-run. While several of these authors are household names in the RL community, this paper was not actually that impressive to me.
The only real "deep" learning here is that they used a GPU library and stochastic gradient descent to perform Q-learning updates on a network with 3 large hidden layers. It was an interesting application paper, but I suspect that the Google acquisition is for something more novel than this work.
We worked on a similar experiment two years ago (deep learning + reinforcement learning algorithm + some innovations, to learn to play Atari 2600 games). We obtained similar scores in the games we tested but we did not submit any paper because we considered that the scores were not good enough. In particular, for Space Invaders, you can easily get 600 points by hiding behind a shelter while continuously firing, and never learn how to avoid the bullet.
So, I was not impressed by their results on Space Invaders.
Overall, we struggled to learn long-term strategies (finding pure reactive strategies is easy) and to learn to avoid bullets. They did too: "The games
Q*bert, Seaquest, Space Invaders, on which we are far from human performance, are more challenging because they require the network to find a strategy that extends over long time scales."
The only real "deep" learning here is that they used a GPU library and stochastic gradient descent to perform Q-learning updates on a network with 3 large hidden layers. It was an interesting application paper, but I suspect that the Google acquisition is for something more novel than this work.