Robots Still Struggle with Household Chores

Every time humanity thinks it is approaching the era of household robots, new research appears showing how much is still left to do. Of course, big progress can be seen in certain areas, but combining all of it into a hardware platform that can work autonomously still seems distant. 

In June 2021, researchers from the University of Bremen presented a paper at ICRA 2021 on the "Household Robot Marathon Experiment" they conducted. In this experiment, the PR2 robot was tasked with first setting a table for a simple breakfast and then clearing the table so that researchers could evaluate the scalability and robustness of mobile manipulation.

It might seem that robots should handle such a task easily, but in fact it remains a serious challenge.

The PR2's task was to prepare breakfast by bringing a bowl, a spoon, a box of milk and a box of cereal to the dining table. After breakfast, the robot had to put washable items into the dishwasher, return the cereal box to its storage place, and throw the milk box into the trash. 

All objects differ in shape and appearance, and the PR2 was given only symbolic descriptions of object locations (in the fridge, on the shelf). The scenario is very realistic but also complex, so the robot needed 90 minutes to complete it.

The researchers were impressed by the good performance of perception, motion planning, grasping and strategy development.  It should be remembered that these were 90 minutes of full autonomy for relatively complex tasks in an environment that is only partly structured and somewhat optimized for the robot. 

In fact, over five trials the robot succeeded five times at the table-setting task. It was not flawless: the PR2 had particular trouble grasping complex objects such as a spoon, but the framework developed by the researchers could successfully recover after each failure by adjusting parameters and repeating the failed action. 

In the long term, failures and the ability to recover from them are even more useful than never failing at all. The cleanup task was more difficult for the robot: during two of the five trials it suffered unrecoverable failures. 

The paper states: "Clearing the table was a more difficult task than setting it, due to the use of the dishwasher and the difficulty of side-grasping objects located far from the edge of the table. In two of the five runs we encountered an unrecoverable error. In one of the runs, because of instability in the grasp trajectory and the robot not tracking it perfectly, the PR2's fingers pushed the milk away during grasping, which led to a very unstable grasp."

“As a result, the box fell to the ground during the transport phase. Although during table setting the robot managed to pick up a tipped-over cup and successfully bring it to the table, lifting the milk box from the floor was impossible for the PR2. Another unrecoverable error occurred when a dishwasher rack got stuck on a PR2 finger. Another serious failure happened when placing the cereal box into a vertical drawer, which was difficult because the robot had to reach very high. When the gripper opened, the box dropped to the floor and was crushed when the drawer closed”

Although the paper focused heavily on these robot failures, it should not be forgotten that its fully autonomous operation for 90 minutes is still impressive. The researchers committed to running such an experiment and placed their robot in a practical environment, so that it performed practical tasks in full autonomy over a long period of time. 

Robots operating in real environments, such as human households, require a huge amount of diverse knowledge to perform their tasks. Such knowledge is difficult to acquire. One natural way to do this is learning from human demonstrations. Nowadays, virtual reality technology is becoming very popular and accessible. 

Virtual reality systems allow people to interact with a virtual environment in a natural and intuitive way. Recorded data on people performing tasks in virtual reality is a powerful source of knowledge about everyday activities that can be used to train robots. 

Virtual reality systems make it easy to change environments and task scenarios, provide high-precision data about real-world events, and give access to the physics of the base world. The researchers found a way to train their robot on human data from virtual reality. Difficulties arose from differences between the virtual and real environments and the objects in them, from the capabilities of humans and robots, and from the context of the tasks. 

There is a lot of research indicating that far more work is needed than we thought for robots to become sufficiently useful in the real world and to handle important tasks such as preparing a table for breakfast.