"To develop robots, you have two options: You can either simulate an environment and robot with software and hope the results are accurate enough that you can load it into a machine and watch it walk."
This is much harder than you'd think.
Here's a fun story from 15 years ago when a friend of mine tried to do some simple AI:
His goal was to have a humanoid shape created is software learn how to walk using simple AI. The idea was that it would obey some basic laws (gravity, the limits of its joints, etc.), do something random, check whether or not it was closer to the goal of walking, tweak its parameters, and iterate. The chosen goal was not to fall over.
He set up the program, let it run over the weekend and let it do millions of tries. Hopefully when he came back to the office it would have learned how to walk, or at least stand up without falling.
His disappointment was huge when he came into the office: The simulated robot was sitting down with its knees bent, thus having achieved the goal of not falling over.
I've heard a similar story about small self driving cars.
The cars would drive around using a random algorithm, then copy and tweak the algorithm of the longest running car when they crashed. The researcher left the room to let the cars work, only to come back and find that each of the cars had deduced that the perfect solution was to remain perfectly still. After all, if they didn't move, they couldn't crash!
That's the story story of a game-learning program that found the best way to "not lose" at tetris was to pause right before a brick extended above the top of the level, and leave it paused indefinitely.
Funny anecdote, but "artificial intelligence" (which people misuse as a fancy term when what they really mean is task optimization) requires setting the right goals. You know, like moving from point A to point B.
Yes, seems like more anecdote than reality. Reality is, you wouldnt just 'code an AI', and leave it running for the weekend, and then act all surprised when it has bugs in. You'd work your way eg through Sutton's tasks, like drive a car up a hill https://en.wikipedia.org/wiki/Mountain_Car , try not to fall off a cliff, http://webdocs.cs.ualberta.ca/~sutton/book/ebook/node65.html balance an arm http://webdocs.cs.ualberta.ca/~sutton/book/ebook/node110.htm... , and so on. And since these mostly learn really quickly, and ones initial implementation will be buggy, you wouldnt go away for a weekend and leave it running, youd just sit there running it for a minute or two, fixing bugs, running again, and so on.
Fortunately, this is not an issue in reinforcement learning any more. It's quite simple to train humanoids in simulators to walk. But that knowledge is completely untransferable to the real world.
This is much harder than you'd think.
Here's a fun story from 15 years ago when a friend of mine tried to do some simple AI:
His goal was to have a humanoid shape created is software learn how to walk using simple AI. The idea was that it would obey some basic laws (gravity, the limits of its joints, etc.), do something random, check whether or not it was closer to the goal of walking, tweak its parameters, and iterate. The chosen goal was not to fall over.
He set up the program, let it run over the weekend and let it do millions of tries. Hopefully when he came back to the office it would have learned how to walk, or at least stand up without falling.
His disappointment was huge when he came into the office: The simulated robot was sitting down with its knees bent, thus having achieved the goal of not falling over.