NeatNet: a system that helps robots learn people's tidying preferences
As robots become more capable and affordable, more people may start bringing them into their homes. That is why many roboticists are trying to build systems that can effectively help people with household chores such as cooking and cleaning.
Researchers from the Robot Learning Lab at Imperial College London have developed NeatNet, an innovative machine learning tool that could allow robots to tidy a home environment according to each user's individual preferences. The model, presented in a paper on arXiv, is based on a new variational autoencoder architecture with graph neural network layers.
“Everyone furnishes their home in a unique and personal way, influenced by aesthetic taste, habits and even cultural background. We developed a method for learning people's preferences for how they arrange their homes, so that a robot can then tidy their home in a personalized way”
NeatNet, a method developed by Dr. Jones and his student Ivan Kapelyukh, allows robots to learn a user's unique tidying preferences simply by observing how they arrange furniture and objects in their home. The robot can then use these preferences as a guide for tidying the user's home, reflecting his or her preferences.
Jones said: "For example, suppose you prefer your desk to be arranged compactly so that everything is easy to reach. You might want the robot to learn this preference. After the robot tidies it, your desk will be arranged in a way that is convenient for you."
NeatNet draws inspiration from recommender systems, machine learning tools used by streaming platforms (such as Netflix, YouTube, Spotify) or other websites to recommend new content to users. Recommender systems work by learning users' preferences based on what content they have watched, listened to or accessed in the past.
The researchers said: "If a new movie has been watched by many users with preferences similar to yours, Netflix may recommend that same movie to you too. This is how these methods provide personalized recommendations."
The new recommendation-like tool for learning tidying preferences was based on Kapelyukh's PhD thesis at Imperial College London, which Dr. Jones supervised. Kapelyukh presented the new paper together with his supervisor at the Conference on Robot Learning (CoRL), held in London from November 8 to 11, 2021.
"So far we have presented our results using computer simulations of the robot and its environment, but in our future work we plan to implement this on a real robot in the real world," said Jones.
Essentially, NeatNet processes scenes that have been organized and tidied by users. From these scenes, it learns the user's tidying preferences, which are represented as a sequence of numbers. Finally, it uses these numerical sequences to arrange any group of objects according to the individual's preferences.
“Because a robot does not know in advance how many objects it will encounter, NeatNet uses a graph neural network for processing. This means that instead of training directly on images of the space, it models it as a graph, where each object is a node (or point) and all nodes are connected together”
Using a graph neural network, NeatNet can also learn relationships between different objects. For example, it can learn that a keyboard and mouse are usually placed next to each other, or that cutlery is placed to the side of plates.
In addition to learning general object relationships, NeatNet learns individual user preferences. For example, the system can learn which side of the plate users typically place their cutlery on, since left-handed and right-handed people may have different preferences.
Dr. Jones and Kapelyukh evaluated their technique in a series of experiments, using room layouts created with a tidying simulator that encoded the preferences of 75 different users. In these tests, NeatNet consistently produced neat and personalized room arrangements.
“We found that tidying rooms in a personalized way was more pleasing to users than simply tidying for everyone. This held true even for simple spaces, and in real homes with hundreds of objects there are many more options for how to tidy each room”
As robots become more common, their ability to perform tasks in ways consistent with individual users' preferences could be especially valuable. In this way, NeatNet may prove particularly useful for improving the performance of home assistants and robots.
“Another interesting finding is that, although the neural network represents the user's preferences as a sequence of numbers, we were still able to find some meaningful patterns. For example, NeatNet chose to group left-handed and right-handed users separately based on how they arranged the dining table. This sheds some light on how the model works internally, which is often hard to determine when using neural networks”
So far, NeatNet has only been tested in simulation, but it has produced promising results. Scientists are currently conducting further research aimed at applying and evaluating their method on real robots.
The researchers said: "We will use the robot's cameras (the "robot's eyes") to determine where objects are in the room. In addition, we will look at how to ensure that the proposed arrangements are always safe according to the laws of physics. For example, a plate should not tip over the edge of the table. We will also study methods that take into account how much time the robot will need to complete this tidying. Instead of tidying the entire house, for example, we might want the robot to tidy up only a few items that are very out of place."
