Optimizing LLM Inference with Speculative Decoding
This article explores speculative decoding, a technique to significantly speed up Large Language Model (LLM) inference by converting underutilized GPU capacity into output. It details how a smaller draft model generates candidate tokens, which are then verified in parallel by the larger target model, reducing sequential forward passes and maintaining output quality. This method addresses the memory bandwidth bottleneck in autoregressive decoding.
Read original on ByteByteGoThe Challenge of LLM Autoregressive Decoding
LLMs generate text one token at a time in an autoregressive manner. Each token generation requires a full forward pass through the model, with the current token's prediction depending on all preceding tokens. This sequential dependency means that generating a long response requires a large number of individual forward passes, directly impacting latency. While KV caches optimize attention computations, the fundamental requirement for one pass per token remains.
Memory Bandwidth Bottleneck
A significant observation in LLM inference is that during token generation, GPUs spend most of their time moving model weights from VRAM to compute units rather than performing actual arithmetic. For a 70-billion-parameter model, this can mean transferring 140 GBs for *each* token. This leads to low compute unit utilization (20-40%) while the memory bus runs near capacity, indicating a memory bandwidth bottleneck. This unused computational capacity is the target for optimization.
Speculative Decoding: Parallel Verification
Speculative decoding addresses the memory bottleneck by leveraging the GPU's ability to evaluate multiple positions in parallel during a single forward pass. Instead of generating one token at a time, a smaller, faster "draft model" proposes several candidate tokens. The larger "target model" then evaluates this sequence of candidates in a *single* parallel pass. This allows the target model to process multiple tokens for the cost of approximately one, effectively converting idle compute capacity into faster output.
- Draft Generation: A small, fast draft model generates K candidate tokens in a rapid sequential loop.
- Parallel Verification: These K candidates are appended to the context and fed to the large target model. The target model performs a single forward pass over the extended sequence, predicting the next token at each of the K candidate positions.
- Acceptance & Rejection: Starting from the left, each candidate is compared to the target model's prediction. Matching candidates are kept. The first mismatch causes all subsequent candidates to be discarded. Crucially, the target model's own prediction at the mismatch point is still used, guaranteeing at least one token per verification pass (worst case) and preserving statistical identity with pure target model output.
Lossless Quality Guarantee
Speculative decoding is designed to produce statistically identical output to the target model running alone. This is achieved through carefully defined acceptance rules that ensure the combined probabilities of kept and replaced tokens exactly match the target model's original probabilities, even with sampling.
Performance Factors: Acceptance Rate
The speedup gained from speculative decoding is primarily determined by the acceptance rate – the fraction of candidate tokens the target model keeps. Higher acceptance rates lead to greater throughput. Workloads generating structured or repetitive output (e.g., code, summarization) typically have high acceptance rates because the next token is easier to predict. Open-ended or creative tasks tend to have lower rates, as the draft model is more likely to diverge from the target model's predictions. Sampling temperature also plays a role, with higher temperatures reducing acceptance.