NVIDIA unveils platform for creating interactive AI avatars
In recent months NVIDIA has steadily promoted its artificial intelligence technology, and now it is clear how all the elements fit together. The company announced Omniverse Avatar, a technology platform for creating interactive AI avatars. Omniverse Avatar combines the company's technologies in voice AI, computer vision, natural language understanding, recommendation engines and simulation technologies.
Avatars created on the platform are interactive characters with real-time ray-traced 3D graphics that can see, speak, hold conversations on a wide range of topics and understand natural spoken intent. The platform opens the door to AI assistants that can be easily customized for virtually any industry.
It can help with billions of daily customer interactions, such as ordering at a restaurant, banking transactions, scheduling personal appointments and making reservations, leading to expanded business opportunities and higher customer satisfaction.
“The era of intelligent virtual assistants has arrived. Omniverse Avatar combines NVIDIA's core graphics, simulation and artificial intelligence technologies to create some of the most complex real-time applications ever built. The use cases for collaborative robots and virtual assistants are incredible and go far ahead”
Omniverse Avatar is part of NVIDIA Omniverse, a virtual world simulation and collaboration platform for 3D workflows, which is currently in open beta and serves more than 70,000 users. In his GTC keynote, Huang shared various examples of Omniverse Avatar: Project Tokkio for customer support, NVIDIA DRIVE Concierge for always-on intelligent in-vehicle services, and Project Maxine for video conferencing.
In the first Project Tokkio demonstration, Huang showed his colleagues taking part in a real-time conversation with an avatar created as a toy copy of himself. They discussed topics such as biology and climate science.
In a second Project Tokkio demonstration, he showed a customer service avatar in a restaurant kiosk that was able to see, talk and understand two people as they ordered vegetarian burgers, french fries and drinks. The demonstrations were run on NVIDIA AI software and Megatron 530B, which is currently the world's largest customizable language model.
In a demonstration of the DRIVE Concierge AI platform, a digital assistant on the central dashboard screen helps the driver choose the best driving mode to arrive on time, then follows his request to set a reminder when the car's remaining range drops below 161 km.
Separately, Huang demonstrated Project Maxine's ability to add modern video and audio features to applications for virtual collaboration and content creation. The company created an avatar from a photo of a woman and used Riva for training, text-to-speech and translation into different languages. The digital double looks and sounds very much like a real person and can even turn its head while maintaining natural eye contact.
Omniverse Avatar uses elements of voice AI, computer vision, natural language understanding, recommendation engines, facial animation and graphics, provided through the following technologies:
- Speech recognition is based on NVIDIA Riva, a software development kit that recognizes speech in multiple languages. Riva is also used to generate human-like speech responses using text-to-speech capabilities.
- Natural language understanding is based on the large language model Megatron 530B, which can recognize, understand and generate human language. Megatron 530B is a pretrained model that can compose complete sentences, answer questions across a broad domain, summarize long complex stories, translate into other languages and handle many areas it was not specifically trained for, with little or no additional training.
- The recommendation engine is provided by NVIDIA Merlin, a framework that lets enterprises build deep learning recommender systems capable of processing large volumes of data to make smarter suggestions.
- Perception capabilities are supported by NVIDIA Metropolis, a computer vision framework for video analytics.
- Avatar animation is based on NVIDIA Video2Face and Audio2Face technologies, 2D and 3D facial animation, and AI-based rendering.
These technologies are combined into one application and processed in real time with NVIDIA Unified Compute Framework. Skills packaged as scalable, customizable microservices can be securely deployed, managed and orchestrated across multiple locations using NVIDIA Fleet Command.
Most of the Maxine development tools are already available. Riva is available in open beta and will be free for "small-scale" use. Larger deployments will depend on the launch of the Riva Enterprise program in early 2022. However, Omniverse Avatar will have to wait longer. While the underlying Omniverse platform is now in open beta, Avatar is "in development" and no launch date has been announced.
