Optimizing Generative AI Video Inference with Asynchronous Frame Pipelining on AWS

Synthesia, an enterprise AI video platform, leverages Amazon EC2 G7e instances for generative AI video inference, specifically for models like latent diffusion that use Variational Auto Encoders (VAEs). A common challenge in such GPU-memory intensive workloads is inefficient GPU utilization due to synchronous operations, where GPU compute stalls while waiting for data transfer and host-side processing of decoded video frames.

The Sequential Decoding Bottleneck

Latent diffusion models perform their diffusion process in a compressed latent space. The final step involves decoding this latent video into human-readable pixel frames using the VAE decoder. To manage resource intensity, videos are typically split into chunks (e.g., 4 consecutive pixel frames per chunk). In a traditional "Sequential Frame Generation Pipeline," after a chunk is processed on the GPU, its pixel frames are synchronously transferred to host (CPU) memory and written to storage. This synchronous copy and write operation prevents the GPU from immediately starting on the next chunk, leading to significant GPU stalls and reduced overall utilization.

Asynchronous Frame Generation Pipeline Design

The proposed "Asynchronous Frame Generation Pipeline" aims to overcome this bottleneck by overlapping GPU computation, device-to-host (D2H) data transfer, and host-side post-processing. This is achieved through several key architectural components:

• CUDA Streams: Two CUDA streams are utilized per device: a default "Compute Stream" for GPU kernels and a dedicated "Copy Stream" for D2H transfers. This allows compute and copy operations to run in parallel.
• Dedicated Worker CPU Thread: A separate CPU thread handles reading chunks from host memory and writing them to file, freeing the main Python thread to focus on launching GPU kernels and scheduling D2H transfers.
• Double-Buffering Strategy: Two in-memory buffers are used on both GPU Memory (VRAM) and Host Memory (RAM). Page-locking is applied to host memory buffers to ensure D2H copies are fully asynchronous. This double-buffering ensures that adjacent chunks can be processed concurrently without data corruption, as compute, transfer, and host processing operate on distinct memory areas.
• CUDA Events for Synchronization: To prevent race conditions and data corruption when buffers are accessed concurrently by different components, CUDA Events are used as synchronization barriers. These events ensure that specific operations (e.g., next chunk decoding) wait for prior dependent operations (e.g., previous chunk transfer completion) to finish.

Architectural Diagram of the Asynchronous Pipeline

The implementation leverages PyTorch and the `torch.cuda.Stream` and `torch.cuda.Event` APIs to manage asynchronous execution and synchronization. The figure below illustrates the interaction between these components.