AMD Hummingbird-0.9B: An Efficient Text-to-Video Diffusion Model with 4-Step Inferencing

Introduction

Text-to-video (T2V) generation excels in creating realistic and dynamic videos from text, becoming a hot AI topic with high commercial value for many industries. However, developing a T2V diffusion model remains a challenge due to the necessity of balancing computational efficiency and visual performance. Most current research focuses on improving visual performance while overlooking model size and inference speed, which are crucial for deployment.

To address this challenge, AMD AI research team proposed reducing the model parameters while boosting the visual performance through a two-stage fine-tuning distillation and reward model optimization. This approach cuts the model parameters from 1.4 billion, as seen in the widely used VideoCrafter2¹ model, to 0.945 billion, the size of AMD diffusion model, enabling high-quality video generation with minimal inference steps. As shown in this blog, the proposed AMD Hummingbird-0.9B T2V diffusion model achieves approximately 23x lower latency compared to VideoCrafter2 on AMD Instinct™ MI250 accelerators. In addition, running the same model on a consumer laptop (iGPU: Radeon^TM 880M, CPU: Ryzen™ AI 9 365) was able to generate a video in 50 seconds¹. Pioneering structural distillation for T2V diffusion models, AMD also introduced a new data processing pipeline for high-quality visual and textual data for model training, demonstrating significant progress of AMD in AI-driven video generation.

Why Build AMD Own Text-to-Video (T2V) Model?

The development of this T2V model is driven by several key factors.

Firstly, efficiency plays a crucial role. Existing open-source T2V models like VideoCrafter2² and CogVideo³ have made significant strides in visual performance, but they do not yet achieve efficient generation on typical GPUs. There is a critical need for smaller models that align with operational constraints while maintaining high performance.

Another important consideration is model size and inference speed. Most current research emphasizes improving visual performance, often at the expense of model size and inference speed, which are crucial for practical deployment. AMD approach focuses on balancing these aspects to ensure both high-quality outputs and efficient operations.

Customization is also a key advantage of building our own model. The proposed structural distillation method allows for adaptable and reward-optimized models, providing users with tailored solutions that meet their specific needs.

Data quality is essential for effective model learning. This novel data processing pipeline helps ensure that the model is trained on top-quality visual and textual data, enhancing its performance and reliability. High-quality training data is critical for the model to generate richer textural details and better understand textual input.

Finally, by open sourcing the training code, dataset, and model weights, AMD supports the AI community. This enables developers to replicate and explore T2V generation on AMD AI platforms, fostering an open and collaborative approach to AI development, and ensuring that the benefits of AI advancements are widely shared.

AMD Hummingbird-0.9B Text-to-Video Diffusion Model

With the above considerations, AMD research team proposed an efficient T2V model along with a training pipeline. This pipeline is designed to be user-friendly, allowing users to build their own models and achieve exceptional performance. Additionally, the style of the generated videos can vary significantly based on the training data. If users wish to generate videos with a specific style, it is recommended using the video data that reflects the desired style.

In specific, there are two key techniques to develop an efficient T2V diffusion model: (1) A data processing pipeline; (2) An innovative model distillation method.

Data processing includes video quality assessment, motion estimation, and prompts re-captioning. Low-quality video data slows loss function convergence during training and reduces performance of the T2V diffusion model. The new data processing pipeline proposed by AMD begins with Video Quality Assessment (VQA) models⁴ to evaluate video quality scores, including aesthetics and compression metrics, as shown in figure 1. Then, a motion estimation module filters out dolly zoom videos from the remaining high-quality ones. For improved textual input quality, it leverages the prior knowledge of a multimodal model (Llama-8B)⁵ for text re-captioning. Experiments demonstrated that using high-quality videos, both visually and textually, enables the T2V diffusion model to generate richer textural details and better understand textual input.

Model distillation helps reduce model parameters for deployment, but it has not been applied to Text-to-Video diffusion models. The key challenge is transferring the knowledge from the original model to a smaller model. To solve this, AMD proposed a two-stage distillation method.

First Stage: the initial step includes creating a smaller model by halving the number of blocks in each layer. It is crucial to preserve structural similarity with the original model while cutting down the blocks, as many experiments have confirmed that this similarity aids prior adaptation and cuts down training time significantly.

Taking VideoCrafter2 with 1.4 billion parameters as an example, the parameters of the reconstruct model are reduced to 0.945 billion. We then use the outputs of the original model as supervision to fine-tune the smaller model. It is observed that the training loss converges quickly due to the similar structure. To further speed up the training process, two empirical acceleration methods are employed: (1) directly learn the video results after classifier-free guidance (CFG) from the original model, and (2) inspired by Consistency Models, we minimize the difference between pairs of adjacent points on a Probability Flow Ordinary Differential Equation (PF ODE) trajectory using numerical ODE solvers. As a result of these methods, the first-stage training takes only one GPU-day on eight AMD Instinct™ MI250 GPUs with 64 GB of memory each.

Second Stage: the next step is to further refine the visual performance of the AMD Hummingbird-0.9B T2V diffusion model by improving aspects like tonal range, deep saturation, finer textural details, and vivid colors. One direct approach would be to collect or synthesize additional video data that showcases appealing visual effects (VFX). For instance, some methods capture video clips from movies and use them as high-quality training data, given their production with advanced cameras and thorough post-production. However, data collection requires substantial time and resources. Additionally, downloading extra data consumes more bandwidth and increases waiting times.

This motivated us to develop a method to reuse first-stage training data for better visual performance.  Inspired by T2V-turbo^6,7, we use reward feedback from a mixture of reward models to gradually extract valuable information from sampled video data by extracting relevant visual information from the existing video data. To improve temporal dynamics and transitions in generated videos, we employ both image-text and video-text reward models. The second-stage training also requires only one GPU-day on eight MI250 GPUs with 64 GB of memory each.

AMD Hummingbird-0.9B Training Details

Training Dataset

The WebVid-10M⁸ dataset is used to train the text-to-video diffusion model. A portion of the high-quality video data is selected using a proposed data processing method for the first-stage training. For the second-stage training, a subset is randomly sampled from this selection.

Training Hardware and Software:

The training was conducted on AMD Instinct™ MI250 accelerator, CDNA™ 2, 362.1 TFLOPs FP16 peak theoretical performance, 128GB HBM2e, 3.2 TB/s peak memory bandwidth, 500W; and with Python 3.8, ROCm™ software 5.6.0, PyTorch 2.2.0, and FlashAttention 2.2.0.

Performance Comparison: Enhanced Visual Quality and ~23X Lower Latency

Why 4-Step Inferencing?

Before validating the AMD Hummingbird-0.9B diffusion model against other diffusion models in terms of the visual performance and inference speed, the AMD research team conducted an experiment to examine the latency by using different inference steps. The visual performance and inference speed under different inference steps are shown as below:

In addition, inference performance was also conducted among these models. Figure 5 showcases the outstanding performance of the AMD Hummingbird-0.9B, achieved ~23X inference speedup compared to VideoCrafter2 on AMD Instinct MI250 accelerator. With impressive speed, efficiency and enhanced visual performance, this model sets a new standard in text-to-video generation, demonstrating significant improvements and capabilities.

To provide clear examples, here is video demonstration showcasing the capability of AMD Hummingbird-0.9B diffusion to generate videos with fine-grained textures and vibrant colors.

Conclusion

The AMD Hummingbird-0.9B T2V diffusion model marks a significant advancement in text-to-video generation, leveraging the capabilities of AMD state-of-the-art hardware and software technologies. The structure distillation method proposed for the T2V diffusion model exemplifies the dedication of AMD to fostering AI innovation, offering exceptional performance and efficiency. Additionally, AMD hardware and software excel in training and inferencing large-scale models like T2V diffusion model, making it a key player in advancing AI and empowering developers to push the boundaries of innovation.

Call to Actions

By open sourcing the training code, dataset, and model weights for the AMD Hummingbird-0.9B T2V diffusion model, we support the AI developer community to accelerate innovation without sacrificing visual performance. You are welcome to download and try this model on AMD platforms. To get more information about the training, inferencing and insights of this model, please visit AMD Github repository to get access to the codes, and visit Hugging Face Model Card to download the model file. As a further benefit, AMD is offering a dedicated cloud infrastructure including the latest GPU instance to AI developers, please visit AMD Developer Cloud for specific accessing request and usage. For any questions, you may reach out to the AMD team at  amd_ai_mkt@amd.com.

Additional Resources:

AMD ROCm AI Developer Hub: Access tutorials, blogs, open-source projects, and other resources for AI development with ROCm™ software platform. This site provides an end-to-end journey for all AI developers who want to develop AI applications and optimize them on AMD GPUs.

AMD Model Blogs: Explore various blogs that cover AMD trained models