Generative AI has been in the spotlight since the release of OpenAI’s ChatGPT late last year, and forecasts suggest the market is expected to grow by a compound annual growth rate of 35.6% from 2023 to 2030. Industry leaders are now discovering how machine learning — and foundation models in particular — can be applied to use generative AI in their organizations to accelerate the development of new technologies and smart devices.

In this post, we’ll cover what foundation models are, why they matter for AI technology, and their substantial computing requirements.

What Are Foundation Models?

Generative AI uses machine learning techniques to generate text, images, videos, or other forms of content. Many of the latest iterations of generative AI are based on transformer neural network architectures because they’re easier to scale to larger model sizes (number of parameters) and train using huge datasets.

A subset of generative AI is a foundation model, which is a large neural network trained on a massive unlabeled dataset to perform a wide range of tasks. These foundation models are usually trained with unsupervised learning, which makes them more scalable because it’s easier to gather large amounts of raw data. This differs from narrow AI models that are trained using supervised learning and labeled data to perform single tasks.

OpenAI’s GPT-4 is an example of a popular foundation model. This is a generative pre-trained transformer (GPT) model that can generate human-like text based on prompts. The GPT architecture and training procedure uses unlabeled data to learn the initial parameters of its neural network, and then supervised learning to adapt these parameters to target tasks.

As foundation models like GPT-4 are becoming widespread, they continue to surprise users and researchers alike. They’re showing “emergent behaviors” or unexpected capabilities that go beyond what researchers initially expected them to be able to do based on their training datasets. Many researchers believe that the larger and more complex these foundation models get, the more likely emergence becomes.

Why Foundation Models Are So Powerful

The primary benefit of foundation models is that they can form the basis for further training for specific use cases or domains. The base level knowledge these foundation models have can transfer from one task to another with only a relatively small amount of additional training or fine-tuning. This is a powerful way to accelerate the development lifecycle for AI applications because the models are pre-trained and organizations do not need to start from scratch.

Large language models (LLMs) are trained on huge amounts of text to perform natural language processing tasks, and they’re also useful as foundation models. Since its release, GPT-3 has been further trained with billions of lines of source code to create OpenAI Codex — a foundation model that understands both human language and code. Microsoft’s Copilot uses Codex to help developers write code using natural language.

Similarly, Stability AI’s Stable Diffusion and OpenAI’s DALL-E are both foundation models for generating images from text descriptions. These are based on diffusion models, which are deep generative models ideal for images, video, and even other use cases like molecular design. In the healthcare industry, researchers at Stanford have already fine-tuned the Stable Diffusion model to generate synthetic medical images and alleviate the gap in training data for medical students.

AI Is Becoming a Foundational Technology

Even before the explosion in popularity for foundation models, AI itself has been becoming a foundational technology. Demand for smart solutions is pushing many technology developers to integrate AI into their products across nearly every industry, but this comes with its own set of computing challenges.

Although generative AI and foundation models are evolving at a rapid pace, they’re also very compute-intensive to train. These models are pre-trained on enormous amounts of data — 45 terabytes in the case of OpenAI’s GPT-3 model. This sheer scale requires powerful hardware, and many companies are turning to NVIDIA’s GPUs to handle generative AI workloads.

The good news is that hardware and AI processing costs continue to fall, with chipsets getting faster and cheaper each year. This increasing level of performance per dollar is making new AI solutions even more economical. That said, new AI-powered solutions still require the right hardware components to achieve optimal performance and extended lifecycles.

Technology developers integrating AI into their solutions should consider partnering with a hardware specialist like MBX Systems. We offer strategic guidance for designing, building, and integrating complex hardware products. Learn more about how MBX and NVIDIA have worked together to fast-track the development of AI workflows in hospitals.