MIT researchers build robot that detects hidden objects

Researchers at the Massachusetts Institute of Technology have developed a picking robot that combines vision and radio frequency (RF) sensing to find and grab objects even when they are hidden from sight. The technology could help in implementing it in e-commerce warehouses. 

In recent years, robots have gained artificial vision, touch and even smell. MIT professor Fadel Adib said: "Researchers are giving robots human-like perception. We are trying to give robots superhuman perception." 

Scientists have developed a robot that uses radio waves, which can pass through walls, to detect hidden objects. The RF-Grasp robot combines this powerful sensing with more traditional computer vision to detect and grab items that might otherwise be hidden from view. 

The progress could one day simplify e-commerce in warehouses or help bots pull a screwdriver out of a messy toolbox. The study was presented in May 2021 at the IEEE International Conference on Robotics and Automation. 

As e-commerce continues to grow, warehouse work is still usually the domain of humans rather than robots, despite dangerous working conditions. Partly this is because robots struggle to find and grab objects in such a crowded environment. Today, the two main obstacles in this area are perception and selection. 

Using only optical vision, robots cannot notice an item packed in a box or hidden behind another item on a shelf, since visible light waves obviously do not pass through walls. But radio waves can. For decades, radio frequency identification has been used to track everything from library books to pets. 

RFID systems consist of two main components: a reader and a tag. The tag is a tiny computer chip that is attached to, or in the case of pets, implanted into, the tracked object. The reader then emits a radio frequency signal, which is modulated by the tag and reflected back to the reader. 

The reflected signal provides information about the location and identity of the tagged object. This technology has become popular in retail supply chains: Japan has aimed to use RF tracking for nearly all retail purchases within a few years. The researchers realized that this abundance of RF could be a boon for robots, giving them another way to perceive. 

According to the scientists, RF is a completely different way of perceiving than vision, so it would be a mistake not to explore what it is capable of. The RF Grasp robot uses a camera and an RF reader to search for and grab tagged objects, even when they are completely out of the camera's view. It consists of a robotic arm attached to a gripping hand. 

The camera is mounted on the robot's wrist. The RF reader is device-independent and feeds tracking information into the robot's control algorithm. As a result, the robot continuously collects both RF tracking data and a visual picture of its surroundings. Integrating these two data streams into the bot's decision-making process was one of the most serious challenges the researchers faced. 

At every moment, the robot must decide which of these streams is more important to focus on. This is not just coordinating eyes and hands, it is coordinating RF, eyes and hands. The robot starts the search-and-retrieve process by sending a ping to the target object's RF tag to determine its location. The sequence is similar to hearing a siren behind you and turning to look and get a clearer picture of where the siren is coming from. 

Equipped with two complementary senses, RF Grasp homes in on the target object. When it approaches and even begins to manipulate the item, vision, which provides much finer detail than RF, dominates the robot's decision-making. 

RF Grasp proved its effectiveness in a series of tests. Compared with a similar robot equipped only with a camera, it could accurately identify and grab the target object with roughly half the total movement. In addition, the device demonstrated a unique ability to "clear" the environment, removing packaging materials and other obstacles in its path to reach the target. 

The researchers say this demonstrates an "unfair advantage" of RF Grasp over robots without penetrating radio frequency sensing. Someday this robot could work continuously in e-commerce packaging warehouses. Its RF sensing could even instantly verify the essence of an item without the need to manipulate it, find a barcode and scan it. 

The team of scientists also envisions potential home applications for the robot, for example, finding the right hex key to assemble an IKEA chair. Or it could help find lost items. The robot could turn into a super Roomba that finds your keys wherever you put them.