Optimizing LLM Inference with Prompt Gisting for Cost and Latency Reduction
Shopify's "Gisting" technique optimizes large language model (LLM) inference by compressing lengthy system prompts into learned "gist" tokens. This method significantly reduces context size, leading to lower latency, increased throughput, and decreased GPU infrastructure costs without sacrificing prediction quality. It represents an architectural optimization at the inference layer for AI-driven applications.
Read original on InfoQ ArchitectureIntroduction to LLM Prompt Gisting
Large Language Models (LLMs) often rely on extensive system prompts to guide their behavior and context. These long prompts consume significant computational resources during inference, impacting latency, throughput, and infrastructure costs, especially in high-volume production environments. Shopify's Gisting technique addresses this by introducing a novel approach to compress these prompts.
What is Gisting?
Gisting compresses a long LLM system prompt into a much smaller set of learned "gist" tokens. These tokens are not a conventional summary but a highly optimized, learned representation that makes the LLM behave as if it had processed the full original prompt.
Architectural Impact and Benefits
- Reduced Context Size: Achieves a significant reduction (e.g., 4:1) in the number of tokens the LLM needs to process for the system prompt.
- Lower Latency: Directly translates to faster Time to First Token (TTFT) and overall end-to-end request latency. Shopify observed TTFT dropping from 438ms to 354ms.
- Increased Throughput: More queries can be processed per second (QPS), with Shopify seeing an increase from 20.2 to 23.4 QPS.
- Infrastructure Cost Savings: By improving efficiency, fewer GPUs are required to handle the same workload, leading to substantial cost reductions.
- No Model Modification: A key advantage is that gisting does not require modifying the LLM's core weights or architecture at inference time; the gist embeddings are simply written into the model's embedding matrix.
How Gisting Works: The Two-Step Training Process
Gisting involves a two-step training process to create the gist tokens, based on the concept of prompt compression and contrastive conditioning:
- Teacher Pass: The LLM is run with the original, full system prompt to generate the "teacher logits" of the desired response.
- Student Pass: The LLM is then run with the shorter, initial gist tokens to generate "student logits."
- Training Gist Embeddings: The gist tokens are iteratively trained to minimize the Kullback-Leibler (KL) divergence between the teacher logits and the student logits. This ensures the student's predictions closely match the teacher's, meaning the gist tokens effectively encode the same information as the full prompt. Once trained, these gist embeddings are directly inserted into the model's embedding matrix, and the new gist tokens are registered in the tokenizer.
Complementary Optimization with Prefix Caching
Gisting is not a standalone solution but complements other LLM optimization techniques like prefix caching. While prefix caching avoids recomputing KV tensors for cached prompt sequences, the model still processes these tensors during decoding. Gisting further reduces this overhead by replacing the long prompt with a shorter sequence of learned tokens, allowing for compounded performance improvements when both are used together. This synergistic approach highlights the importance of combining various optimizations in real-world LLM inference systems.