Meta AI researchers create ReSkin, a universal 'skin' for AI tactile perception research
Our sense of touch helps us navigate the world around us. With it, we can gather information about objects (whether they are light or heavy, soft or hard, stable or unstable) that we use to carry out everyday tasks, from putting on shoes to preparing food. Artificial intelligence effectively combines senses such as sight and sound, but touch remains a persistent challenge.
This is linked to limited access to tactile data in the wild. AI researchers who hope to incorporate touch into their models struggle to harness the richness of sensory perception the way humans do. Facebook announced ReSkin, a new open-source sensor "skin" created by Meta AI researchers in collaboration with Carnegie Mellon University, which could help quickly and at scale improve the tactile abilities of AI.
Using advances in machine learning and magnetic sensing, ReSkin offers a low-cost, versatile, reliable and replaceable solution for long-term use. The skin uses a self-supervised learning algorithm that helps automatically calibrate the sensor, making it universal and able to share data with other sensors and systems.
The company plans to release the design, accompanying documentation, code and base models to help AI researchers use ReSkin without having to collect and train their own datasets. This should help accelerate the rapid, large-scale development of AI tactile abilities. Sensor skin such as ReSkin will provide a source of extensive contact data that could be useful in advancing AI across a wide range of sensory tasks, including object classification, proprioception and robotic grasping.
AI models trained on tactile perception skills will be able to perform many types of tasks, including those requiring high sensitivity, such as work in medical facilities, or greater dexterity when maneuvering small, soft or delicate objects. ReSkin can be integrated with other sensors to collect visual, audio and tactile data outside the lab and in unstructured environments. Combining multimodal datasets helps build physically realistic models of the world and use redundancy for self-supervised learning.
ReSkin costs less than $6 per 100 units and even less in large quantities. It is 2-3 mm thick and can be used for more than 50,000 interactions. The skin's characteristics make it ideal for various form factors, such as robot hands, tactile gloves, sleeves and even dog shoes. All of this will help researchers collect tactile data for new AI models that were previously difficult or impossible to gather. ReSkin can also deliver high-frequency triaxial tactile signals for fast manipulations such as sliding, throwing, catching and clapping. When it wears out, it can be easily removed and replaced.
Because ReSkin is an elastomer with embedded magnetic particles, when it deforms in any way, the surrounding magnetic signal changes. These changes can be measured with nearby magnetometers and control methods can be used to convert this data into information such as contact location and the magnitude of applied force.
Currently, most tactile perception experiments rely on a single sensor, because the model had to be retrained each time the skin was replaced, which is inefficient and impractical. To avoid having to train new skin every time it is replaced, researchers worked on creating a universal skin.
"With ReSkin, we overcome these challenges using three key ideas. First, ReSkin eliminates the need for an electrical connection between the soft material and traditional measurement electronics. Magnetic signals depend on distance, so the electronics only need to be nearby. ReSkin is also more efficient than conventional soft sensors because it separates the internal electronic circuit from the passive interface. This makes replacing worn skin as simple as peeling off and sticking on a sticker. Second, instead of relying on input from a single sensor, we could better train the model's mapping function using data from several sensors. In this way, we trained the model on a greater variety of data, which helps it produce more efficient and generalizable results," said the Meta AI researchers.
Instead of collecting calibration data for each new sensor, advances in self-supervised learning are used to automatically fine-tune sensors using small amounts of unlabeled data. ReSkin opens up a wide range of universal, low-cost and scalable tactile sensing modules that are not possible with existing systems.
“Reliable tactile sensing is a serious weak point in robotics. Current sensors are either too expensive, have low resolution, or are simply too bulky for specialized robots. ReSkin may solve some of these problems. Its light and small form factor makes it compatible with arbitrary grippers, and I look forward to further exploring this sensor on robots in our lab”
To show how ReSkin can help researchers advance AI using diverse tactile data that was previously difficult to collect, its work was demonstrated in several application examples:
In-hand manipulation. ReSkin offers useful tactile recognition properties for in-hand manipulation, for example for developing AI to teach robots to use a key and open a door, or to grasp fragile items such as grapes or blueberries.
Measuring tactile forces in the wild. The compact and unobtrusive design makes ReSkin ideal for measuring tactile forces in wild conditions. To demonstrate, researchers placed one magnetic skin and a flexible circuit board inside the sole of a dog shoe. The sensor tracks the magnitude and direction of applied force while the dog rests, walks and runs.
Human-object interaction. ReSkin is useful for measuring forces during natural human interaction with objects, for example when we lift items or push them with our hands. Researchers placed the skin and a rigid circuit board on the index finger of a test subject's right hand under a nitrile glove.
Contact localization. ReSkin can be scaled to localize contact over large areas. This is useful for building models based on knowledge of an object's location. For example, if a researcher wants to teach a robot to reliably grasp something, it needs to know where it is touching the object and with how much force.
