Can anyone that works on this technology provide an honest assessment of where this technology actually stands? How much instrumentation is actually required, what the interaction quality is, how much trouble do humanoids have with in the wild daily tasks like turning doorknobs, recovering from falls, avoiding knocking into things, etc.
I work in this field, and I wrote my bachelor's thesis here, not with humanoids but with VLAs (think chatgpt connected to a robot arm)
It's certainly not there yet for anything practical, there's also certain bits and structures that don't have accurate names during construction, and it is important to keep that in mind - so a robot is unlikely to understand what it means to say "put the left bit of this box onto this right bit" due to ambiguity, a human would understand that
Plus we have no good reliable accuracy testing data in most cases (most tests occur on a few demos, but that isn't a good representation of how must things work), popular benchmarks, such as libero have been saturated, and nearly everything gets 95% there, most companies and researchers have their own benchmarks here.
Plus companies lie alot, and do very dangerous things in thier videos, I.e. these robots should not be standing very close to humans, because of being dangerous.
There are also legitimate concerns of misuse of these robots that need to be accounted for, misuse does not have to be warfare, but can be as simple as confusing it while it is cutting tomatoes with a knife.
Turning doorknob is easy, and fail recovery is also being worked on, but we don't have reliable statistics anywhere on that. The hard part is on practical things, as in when placing bricks or attaching a part during manufacturing it needs to ensure that it is aligning everything correctly....and that's hard, while it is impressive, it is very irresponsible to keep humanoids at home (people are irresponsible when untrained), for example, lawnmowers injure about 6400 people a year...and that is not an everything machine.
Humanoids in general are...not appealing in specific, due to maintainable of joints, complexity, but robot arms in particular, expecially on wheels (check mobile aloha), are likely to be able to do tasks such as clean up in hotels, after a guest had left, or replace some cooks in restaurants (if their work is consistent)
This is absolutely not true check out Sunday robotics and their robot that folds clothes. Sunday robotics says it does holding of clothes correctly 99% of the time that is real world usage today. They even have three hour video of the robot folding the clothes.
No, the above is absolutely exactly true. Robots folding clothes and tying shoelaces etc is nothing but a tech demo at this point.
It's difficult to grok this because if you watch a human folding a t-shirt, you can reliably predict that the same human will fold a different t-shirt just as well, and in fact be perfectly capable of folding a wide variety of other clothes items as well. Not so for robots. With robots, what you see is precisely what you get. If you see a robot folding a t-shirt, all that means is that that particular robot can fold that particular t-shirt. The state of the art today is that the same robot cannot be expected to be able to fold a different t-shirt.
For example, see this article about Mobile ALOHA at Google. There's a passage where the visiting, awe-struck, journalist asks whether the robot he's just seen tying up a pair of shoelaces can tie up his own shoe.
“If I gave it my shoe,” I ventured, “would it just totally fail?”
“We could try,” Tompson said. I removed my right sneaker, with apologies to
anyone forced to handle it. Tompson gamely placed it on the table, while
Driess reloaded the policy.
“To set expectations,” Driess said, “this is a task that is thought of as
being impossible.”
Tompson eyed his new experimental subject with some trepidation. “Very short
shoelaces,” he said.
The policy booted up, and the claws set to work. This time, they poked at
the shoelace without getting a grip. “Do you give consent for your shoe to
be destroyed?” Driess joked, as the hands grabbed at the tongue. Tompson let
them try for a few more seconds before hitting the Failure pedal.
As to ACT-2 which basically uses the same techniques as ALOHA (imitation learning) far as I can tell, that's a commercial product and the information they give on their site is difficult to parse. E.g. they say they have 99.1% ±0.3 success rate, 778 successful folds and 9 garment types which is low enough to engender some trust they're not trying to inflate their numbers, but they don't say whether they trained on the garments used in evaluation or not. Chances are they did, because that's the current limit of the technology, i.e. if the garment being folded is unseen (as opposed to the environment, which they tout) then performance is basically random. So either ACT-2 have a major breakthrough that is a few leaps and bounds away from the current state of the art, or you've just watched a tech demo.
The fact that they only advertise "9 garment types" though is a big hint: they have the same problem with generalisation as everybody else at this point in time.
You make some very good points about the ACT-2 if it’s nine garment types that’s still very acceptable but the point you bought up about them training on exactly those pieces of garments is a possibility.
Cheers. You can see that's what they did if you look at the 4th video on the page, under the figure titled "Quality Remains High Across Garment Types" right below the paragraph that starts with "To put these scores in context". Sorry, I have no idea how to link to that video specifically.
In the left half of the video you can see that the robot is (trying to) exactly match the folds of the human in the right half and it's doing so while folding the exact same garments on the exact same surface.
The right half of the video is not a training demonstration, I don't think, since the robot is trained by teleoperation AFAICT (it needs to because it must use its head-mounted camera to control its movements) but that just underlines the degree to which their training regime is exactly copying the movements of a trainer, on the same garment, in the same environment.
This is a limitation of the training approach, by RL. With RL you learn a mapping between sets of pixels (as in the video that comes in through the robot's camera) and robot actions (as in actuator commands). What that means is that once a policy is trained and the robot is deployed, if the input pixels are significantly different than the input pixels at training the robot doesn't have a policy that matches the input pixels and so it can't find the right actions to take. So they have to keep the training and deployment garments and even the environments the same, or as similar as possible.
You can see some more evidence of this in the video right under the paragraph with the title "Hill-Climbing Reliability Through Post-Training". The robot at the front of the video, with the bright red trim, is shown trying to fold a grey t-shirt with white flower decorations and a frilly hem (how adorable <3). But you can tell it fails because the video stops before the robot has completed the fold. If it could complete it, you can rest assured that the video would be showing off the entire folding sequence as it does for the robot with the green cap on the other side of the bed.
That paragraph is making a claim about a "post-training" regime that's supposed to improve generalisation but it leaves more details to a "separate technical post". So I can't tell what it's supposed to be doing, but I don't think it's working.
When I watch videos like that I always remind myself that a) I'm watching a tech demo created to attract investment and b) I've watched way too many of those, going all the way back to the Boston Dynamic videos of Robot Dog or of Atlas doing backflips and yet the state of the art hasn't really budged since. Such videos make it easy to overestimate the state of the art in autonomous robotics and in fact are meant do precisely that: play up robots' true capabilities. It's just impossible to say anything about a robot's general capabilities by watching a few minutes or even a few hours of video. OtoH if you know what to look for you can tell everyone is basically stuck at the same level and trying the same things to escape it. The truth is robotic autonomy is several major breakthroughs away and nobody has any idea how to get there. So we'll be seeing many more of those tech demo videos in the years to come.
Unfortunately I can't go into too much depth but I've seen many first hand evaluations of these robots- I'm a robot technician with first hand experience working on these robots. There is an incredible amount of effort and money going into training on many different tasks, including a few you mentioned and many more. They're getting pretty good, but don't expect to see this generation of bots doing their thing in public for a while, if ever. Generally they're pretty safe but they're a long way from safe enough to be sold to individuals in a household environment. I expect to see an incredible amount of progress in the next couple years.
IMHO the real test is if any robotic startup currently selling (or planning to sell) robots as a service for homes not just use it but gets returning users from it.
I did professionally few prototypes with robots and progress is real yet very far from what the average customer would find reliably useful in menial tasks.
FWIW I do think https://rodneybrooks.com/why-todays-humanoids-wont-learn-dex... remains relevant, namely dexterity is also a hardware problem, grippers aren't hands. They even clarify "multi-finger dexterous manipulation remains challenging." and those aren't even fingers with a lot of sensors.
There is a LOT of monkey business when it comes to robotics industry, a lot of demos you see are well crafted for that specific demo, a lot of times there’s even actual fraud “like claiming it’s an autonomous but it was actually teleoperated during the demo”, a lot robots that make most hypes are practically useless (dog robots or qugv for example) no matter how they try to create a usecase, when it comes to practicality, most fail. Robotics as an industry is niche (not to be confused with cobotics) the only exception is drones which is something I said in a workshop back in 2020 that drones are the only potential and will dominate if we solved the endurance issue, which is also why they immediately got weaponized because humans are usually evil and will utilize useful stuff in bad things.
Humanoids are far from being remotely useful in real life scenarios, yet.
Fun fact: most (if not all) qugv can’t go reverse on a stairway.
