Meta to Build AI Research SuperCluster, the World's Largest AI Research Supercomputer
New-generation advanced AI will require powerful new computers capable of performing quintillions of operations per second. Meta announced that it will design and build the AI Research SuperCluster (RSC).
It is one of the fastest AI supercomputers operating today, and it will become the fastest AI supercomputer in the world once fully built in mid-2022. The company's researchers have already begun using RSC to train large natural language processing (NLP) and computer vision models for research, with the goal of one-day training models with trillions of parameters.
RSC will help Meta's AI researchers build new and better AI models that can learn from trillions of examples, work across hundreds of different languages, seamlessly analyze text, images and video, develop new augmented reality tools, and much more. Company employees will be able to train the largest models needed to develop advanced AI for computer vision, NLP and speech recognition.
"We hope RSC will help us build entirely new AI systems that can, for example, enable real-time voice translation for large groups of people, each speaking a different language, so they can collaborate seamlessly on a research project or play an augmented reality game together. Ultimately, the work done with RSC will pave the way for building technologies for the next major computing platform — the metaverse, where AI-based applications and products will play an essential role," Meta said.
“The experiences we're building for the metaverse require enormous compute power (quintillions of operations per second!), and RSC will enable us to build new AI models that can learn from trillions of examples, understand hundreds of languages, and more”
Old system: 22 000 Nvidia V100 GPUs. Today: 6,080 Nvidia A100 GPUs. Mid-2022: 16,000 Nvidia A100 GPUs.
Compared with the AI research cluster Meta uses today, which was built in 2017, RSC changes the number of GPUs involved, how they interact, and the storage connected to them.
"In early 2020, we decided that the best way to accelerate progress was to design a new computing infrastructure from scratch to take advantage of new GPU technology and network fabric. We wanted this infrastructure to be able to train models with more than a trillion parameters on datasets up to an exabyte in size, which is equivalent to 36,000 years of high-quality video to give a sense of scale," the researchers said.
The old system connected 22,000 Nvidia V100 GPUs with tensor cores. The new one switched to the Nvidia A100, which dominated recent AI system performance benchmarks.
The new system currently consists of a cluster of 760 Nvidia DGX A100 computers with 6,080 GPUs. The computing cluster is linked via an Nvidia InfiniBand network running at 200 gigabits per second. Storage includes 46 petabytes (46 million billion bytes) of cache memory and 175 petabytes of bulk flash memory.
Compared with the old V100-based system, RSC demonstrated a 20-fold speedup on computer vision tasks and a threefold speedup on large-scale natural language processing workloads.
When the system is completed in mid-2022, it will connect 16,000 GPUs, which the researchers say will make it one of the largest of its kind. At that point, its cache and storage will have a capacity of 1 exabyte (1 billion billion bytes) and will be able to deliver 16 terabytes of data per second to the system.
"We expect this step change in computing capability will allow us not only to build more accurate AI models for our existing services, but also to deliver entirely new user experiences, especially in the metaverse," Meta engineers Kevin Lee and Shubho Sengupta said.
The new system will also focus on reliability. This matters because very large networks can take weeks to train, and you don't want a failure halfway through a task that forces you to start over.
It's worth noting that the largest production-ready system tested in the latest round of MLPerf neural network training benchmarks was a 4,320-GPU system presented by Nvidia. That system was able to train the BERT natural language processor in less than a minute. However, BERT has only 110 million parameters compared with the trillions Meta wants to work with.
The launch of RSC is also tied to a change in how Meta uses data for research.
"Unlike our previous AI research infrastructure, which used only open-source and other publicly available datasets, RSC also helps us effectively put our research into practice by allowing us to include real-world examples from Meta's production systems in model training," Meta said.
The researchers said RSC will take additional precautions to encrypt and anonymize this data to prevent leaks. These steps include isolating RSC from the broader internet, with no inbound or outbound connections.
Traffic to RSC can only come from Meta's production data centers. In addition, the data path between storage and GPUs is fully encrypted, and the data is anonymized and subject to verification to confirm anonymization.
