MIT Researchers Develop AI Algorithm That Understands Relationships Between Objects
When people look at any surface, they see objects and their relationships. On your desk there may be a laptop that is to the left of a phone, which is in front of a computer monitor.
Many deep learning models struggle to see the world this way because they do not understand the tangled relationships between individual objects. Without knowing these relationships, a robot designed to help someone in the kitchen would have difficulty carrying out a command such as "take the spatula that is to the left of the stove and place it on the cutting board."
In trying to solve this problem, researchers at the Massachusetts Institute of Technology developed a model that understands the basic relationships between objects in a frame. Their model represents individual relationships one at a time, then combines these representations to describe the whole frame. This allows the model to generate more accurate images from text descriptions, even when the scene includes several objects that are in different relationships to one another.
This work could be applied in situations where industrial robots must perform complex, multi-step manipulation tasks, such as stacking items in a warehouse or assembling equipment. It also brings the field one step closer to machines that can learn and interact with their environment the way people do.
“When I look at a table, I can't say that there is an object at location XYZ. Our brain doesn't work that way. In our minds, when we understand a scene, we really understand it based on the relationships between objects. We think that by building a system that can understand relationships between objects, we could use it to manipulate and change our environment more effectively”
The framework developed by the researchers can generate an image of a scene based on a text description of objects and their relationships, for example: "A wooden table to the left of a blue chair. A red couch to the right of a blue chair."
Their system broke these sentences down into two smaller parts, each describing an individual relationship ("a wooden table to the left of a blue chair" and "a red couch to the right of a blue chair"), and then modeled each part separately. These parts are then combined through an optimization process that produces an image of the scene.
The researchers used a machine learning method called energy-based models to represent the relationships between individual objects in the description of the scene. This method allows them to use one energy-based model to encode each relational description, and then compose them together so that all objects and relationships are inferred.
"By breaking sentences into shorter parts for each relationship, the system can recombine them in different ways to better adapt to scene descriptions it has never seen before," said CSAIL PhD student Shuang Li.
“Other systems treat all relationships as a whole and generate an image from the description in one go. However, such approaches fail when we have out-of-distribution descriptions, such as descriptions with a large number of relationships, because this model cannot truly adapt a single snapshot to create images containing more relationships. However, when we compose these separate, smaller models together, we can model a larger number of relationships and adapt to new combinations”
The system also works in reverse: given an image, it can find text descriptions that match the relationships between objects in the frame. In addition, the model can be used to edit an image by rearranging objects so that they match a new description.
The researchers compared their model with other deep learning methods that received text descriptions and created images showing the corresponding objects and their relationships. In every case, their model outperformed the baselines.
They also asked people to evaluate whether the generated images matched the original description. In the most complex examples, where the descriptions contained three relationships, 91% of participants concluded that the new model performed better.
“One interesting thing we found is that for our model, we can increase our sentence from one description to two, three, or even four descriptions, and our approach still generates images that are correctly described by the relationships, while other methods fail”
The researchers also showed the model images of scenes the system had not seen before, along with several different text descriptions of each image, and they were able to successfully identify the description that best matched the relationships between objects in the image.
When the researchers gave the system two descriptions of relational scenes that described the same image but in different ways, the model was able to recognize that the descriptions were equivalent. The scientists were impressed by the reliability of their system, especially when working with descriptions it had not encountered before.
“This is very promising, because it is closer to how people work. People can see only a few examples, but we can extract useful information from just those few examples and combine them to create an infinite number of combinations. And our model has a property that allows it to learn from less data but generalize to more complex scenes or image generations”
Although these first results are encouraging, the researchers would like to see how their model performs on more complex real-world images with noisy backgrounds and objects that block one another.
They are also interested in eventually incorporating their model into robotic systems, which would allow a robot to identify relationships between objects and then use this knowledge to manipulate objects in the world.
“Developing visual representations that can deal with the compositional nature of the world around us is one of the key open problems in computer vision. This research has significantly advanced the solution of this problem by proposing an energy-based model that explicitly models multiple relationships between objects depicted in a scene. The results are truly impressive”
