Facebook will train robots to take out the trash and unload the dishwasher
In summer 2021, Facebook announced a new step in the company's push toward what it calls "embodied AI" — technology that will one day allow robots to perform everyday tasks, such as stocking the fridge with groceries or taking out the trash.
The company unveiled Habitat 2.0, an upgrade to its habitat simulation platform that lets researchers train their robots at high speed in virtual environments that render, with maximum accuracy, the minute details machines may encounter in everyday settings (for example, in a kitchen or living room).
By letting robots move freely through simulated Habitat environments, researchers gain obvious savings in money and time compared with training them in the real world. Facebook hopes the platform will speed up the development of assistive robots that help people with household chores.
For many years Facebook has been working to advance "embodied AI," which focuses on training general-purpose robots to move through the real world alongside people, responding to previously unknown environments or unexpected objects in their path.
This could mean home robots that bring their owners items on command and load the dishwasher. It would also open the door to more serious technologies, such as a guide robot helping visually impaired people walk outdoors. To help machines perform useful tasks in the real world, they need experience in hundreds of different real environments — with children's toys scattered on the floor and rugs with flipped corners.
This is where simulation can be a game changer. Facebook's scientists believe that rather than moving a physical robot through different apartments, houses and offices over many months and years, a far more pragmatic approach is to expose AI to virtual environments to speed up training. Two years ago the company introduced the first version of Habitat, which included the Replica dataset — a compilation of 18 3D scans of spaces (from office conference rooms to two-story houses).
Replica was an ultra-realistic library of settings that showed some of the finest details in each location — from mirror reflections to rug textures. However, it was a static dataset, meaning a robot could move through the virtual space but could not interact with any of the objects.
With Habitat 2.0, the company set about solving this problem. Now robots can not only move through the new platform's virtual environments but also interact with objects they might find in a typical kitchen, dining room or other space used by people.
“Thanks to this new dataset and platform, AI researchers can go beyond simply building virtual agents in static 3D environments and move closer to creating robots that can easily and reliably perform useful tasks, such as stocking the fridge, loading the dishwasher or delivering items on command”
Replica also received an update called ReplicaCAD, which supports moving and manipulating objects. ReplicaCAD is the result of more than 900 hours of work by 3D artists, who created 111 unique living-space layouts, including 92 objects. Particular attention was paid to material composition, geometry and surface texture of the items, as well as whether the items have special mechanisms, such as refrigerator doors that open and close.
The artists even recreated realistic clutter such as kitchenware, books and furniture. The platform is still far from perfect: for example, ReplicaCAD does not support non-rigid dynamics such as liquids, fabrics and ropes, and it does not include sound or tactile recognition. Nevertheless, Habitat 2.0 allows robots to virtually perform many tasks, from setting the table and cleaning the fridge to tidying up the house and other ordinary household chores.
Facebook is not the only company interested in virtual simulation of "embodied AI." For example, in 2021 the Allen Institute in Seattle released ManipulaTHOR, a sophisticated virtual robot hand capable of manipulating objects in more than 100 simulated environments. However, Facebook may score extra points in this contest, since Habitat 2.0 runs twice as fast as most 3D simulators currently available to researchers.
“Such speed significantly cuts time, allowing researchers to run experiments that normally take more than six months in just two days. Shorter experiment times will let researchers test new ideas much faster and more often, leading to more simulations completed and paving the way for better progress in the field”
Facebook's research team still has a long way to go before Habitat reaches its full potential. Speed can always be improved, but there is another key challenge — linking different things together. Although robots can today be trained on individual skills, such as picking up, placing or opening a drawer, they still must be taught how to perform many of these tasks and chain them together without accumulating errors, which still significantly limits the usefulness of the technology.
Facebook also aims to expand ReplicaCAD and include more living spaces reflecting different cultures and types of objects. To that end, the company opened Habitat to third-party 3D assets and announced a partnership with Matterport, a company specializing in spatial data.
