MIT researchers create system for designing and training intelligent soft robots
Imagine you want to build the best stair-climbing robot in the world. You would need to optimize both the brain and the body, perhaps by giving the bot several high-tech legs and feet combined with a powerful climbing algorithm.
Although the physical body design and its brain ("control") are key components that enable a robot to move, existing benchmark environments favor only the latter. Optimizing both elements together is difficult: it takes a lot of time to train different robot models to do different things even without a design element.
Scientists from the Massachusetts Institute of Technology's Computer Science and Artificial Intelligence Laboratory (CSAIL) set out to fill the gap by developing "Evolution Gym", a large-scale testing system for jointly optimizing the design and control of soft robots, drawing inspiration from nature and evolutionary processes.
Robots in the simulator are somewhat like soft movable Tetris pieces, made up of soft, rigid and actuated "cells" on a grid, which are used for walking, climbing, manipulating objects, changing shape and navigating dense terrain layers. To test the robots' abilities, the team developed its own co-design algorithms by combining standard design optimization methods and deep reinforcement learning (RL) methods.
The co-design algorithm works as a power couple, where design optimization methods evolve the robot bodies, while RL algorithms optimize the controller (the computer system that connects to the robot to control its movements) for the proposed design. Design optimization asks, "How well does the design work?", while control optimization gives a score that might read as "five for walking".
The result resembles a small Olympics for robots. In addition to standard tasks such as walking and jumping, the researchers also included some unique tasks, such as climbing, flipping, balancing and climbing stairs.
Across more than 30 different environments, the bots performed well on simple tasks such as walking or carrying an object, but they failed on more complex conditions such as catching and climbing stairs, revealing the limitations of current co-design algorithms. For example, sometimes the optimized robots displayed what the team calls "frustration", clearly suboptimal behavior across many tasks. For instance, a catcher robot often dove forward to catch a block falling behind it.
Although the robot designs evolved autonomously from scratch, without any prior knowledge of co-design algorithms, moving toward more evolutionary processes, they often came to resemble existing natural creatures, outperforming robots built by hand.
“By creating Evolution Gym, we aim to push the boundaries of machine learning and artificial intelligence algorithms. By building a large-scale benchmark focused on speed and simplicity, we not only create a common language for sharing ideas and results in reinforcement learning and co-design, but also give researchers the ability to use modern computing resources to contribute to algorithm development in these areas. We hope our work brings us one step closer to a future with smart robots like you or me”
In some cases, when robots learn the way humans do, trial and error can lead to a better understanding of the task underlying reinforcement learning. Robots learned to perform a task such as pushing a block by receiving some information that helps them: for example, "seeing" where the block is and what the nearby terrain looks like.
The robot then receives some score of how well it performs (a "reward"). The more the robot pushes the block, the higher the reward. The robot must simultaneously balance exploration (perhaps asking itself, "Can I increase my reward by jumping?") and exploitation (further learning behavior that increases the reward).
The different combinations of "cells" that the algorithms came up with for different designs were very effective: one turned into a jumping horse-like shape with leg-like structures mimicking what is found in nature. The climbing robot developed two arms and two leg-like structures (like a monkey) to help it climb. The lifting robot resembled a two-fingered gripper.
One direction for future research is so-called "morphological development", where a robot gradually becomes smarter as it gains experience solving more complex tasks. For example, you would start by optimizing a simple walking robot, then take the same design, optimize it for carrying, and then for climbing stairs. Over time, the robot's body and brain "transform" into something that can solve more complex tasks compared with robots trained directly on the same tasks from the start.
“Evolution Gym is part of a growing recognition in the AI community that body and brain are equal partners in supporting intelligent behavior. There is still much to be done to figure out what shapes this partnership. The gym will likely become an important tool for working on such questions”
Evolution Gym is open source and free to use. This was done deliberately, as the researchers hope their work will inspire the creation of new and improved coding algorithms.
