Meta Researchers Build AI System That Brings Children's Drawings to Life
Children draw incredibly unique and unusual characters that spark our imagination and make us think a little differently in order to recognize people and objects in their drawings. While it may be fairly easy for a parent or teacher to see what a child's drawing is meant to show, AI cannot handle this task.
Children's drawings are often created in abstract, quirky ways, so if a figure's legs are placed unsteadily or both hands are on one side of its body, this confuses even modern artificial intelligence systems that excel at detecting objects in photorealistic images and drawings.
Meta researchers are working to overcome this challenge so that AI systems can better recognize drawings of human figures in the many different ways children create them. They announced the first-of-its-kind method for automatically animating hand-drawn children's figures of people and human-like characters (with two arms, two legs, and a head).
These drawings can be brought to life in just minutes with artificial intelligence. By uploading them to the system prototype, parents and children can experience the excitement of watching their drawings turn into moving characters that dance and jump. They can also download their animated drawings to share with friends and family. With parents' permission, they can also submit these drawings to improve the AI model.
"By teaching AI to work effectively with this typical form of human creativity, we hope this project brings us closer to creating AI that can understand the world from a human perspective. We also hope this work will spur further research into using AI to enhance human creative abilities and inspire new creative ways to use this technology," Meta said.
The researchers' goal was to create an artificial intelligence system that can identify and automatically animate human figures in children's drawings with a high degree of success and without any human intervention.
While many AI tools and methods are designed to process realistic images of people, children's drawings add a layer of diversity and unpredictability that makes it much harder to determine what is depicted. "People" in children's drawings come in all kinds of shapes, colors, sizes and scales, but they have little in common when it comes to body symmetry and point of view.
The scientists approached this task using a four-stage process, fine-tuning their approach at each stage to adapt to the enormous variety found in children's drawings.
The researchers developed a system for training AI to recognize a human-like figure in a child's drawing, which can be difficult when other objects or shapes on the page have similar features, colors or sizes. After that, the human figure is isolated, and nothing is added to the page so the animated figure stays as close as possible to what the child intended. Then the figure's joints are identified so the animation can move appealingly. Once all these steps are complete, the drawing is ready for animation.
To bring 2D figures to life using 3D motion capture, the team takes advantage of the fact that many children draw using what scientists call curved perspective. Many children typically draw body parts first from their most recognizable point of view, which may differ from how they look on real people. For example, they usually draw legs and feet from the side, and heads and torsos from the front.
The researchers use this perspective during the motion retargeting stage. Regardless of the lower and upper body, they automatically determine whether the movement is more recognizable when viewed from the front or the side. Using the selected views, the movement is projected onto a single 2D plane and used to control the character.
"By sharing our work, we also hope to encourage more computer vision work in the area of amateur drawings. Further research on this project could focus on identifying and applying more tailored movements to figure subcategories, such as superheroes, princesses, monsters and ninjas," the researchers said.
A more detailed analysis of the character's parts could also help identify antennae, tails and capes and apply secondary motion elements to make the animation more appealing. Someday, an AI system may be able to take a complex drawing and instantly create a detailed animated cartoon, using several fantastical characters interacting with each other and background elements.
With AR glasses, these stories could even come alive in the real world, dancing or talking with the child who drew them just moments ago. The possibilities are endless, just like human imagination.
If adults consent to their child's drawing and their interactions with the demo being used for further research, the drawing and interactions may be used, released, or published for any of the following purposes:
- To teach the model to better identify, segment, and place joints on hand-drawn characters.
- To provide high-quality viewer annotations.
- As part of academic research work or video.
- As part of a publicly available database for researchers developing creative tools for children.
- As part of future initiatives to encourage collecting additional images.
