Implementing Caching Strategies for LLM Workloads to Reduce Cost and Latency

The article highlights that LLMs can incur repeated costs for identical or semantically similar queries. It proposes a caching architecture to mitigate this, drawing lessons from caching practices in traditional data pipelines where redundant computations often go unnoticed until cost reviews. The primary goal is to determine when work has already been done and reuse previous results, thereby reducing compute and improving latency.

Multi-Tiered Caching Architecture

A robust LLM caching solution often involves a hybrid approach combining exact-match and semantic caching. This tiered strategy optimizes for both performance and flexibility:

• Tier 1: Exact Match Cache: Utilizes cryptographic hashing (e.g., SHA-256) of normalized request bodies (query, context, model settings, caller scope) as keys in an in-memory store like Redis. This provides O(1) lookup for identical requests, offering the fastest and cheapest hits.
• Tier 2: Semantic Match Cache: For queries that are similar but not identical, this tier uses an embedding model to convert the query into a vector. These vectors are stored in a vector database, allowing for similarity searches (e.g., cosine similarity). A configurable threshold determines what constitutes a "close enough" match, which needs careful tuning to balance accuracy and hit rate.
• Hybrid Approach: The recommended strategy is to check the exact-match cache first. If it's a miss, a semantic search is performed. If a semantic match is found, the result is then promoted and stored in the exact-match cache under the hash of the new query, effectively 'teaching' the exact cache about paraphrases.

def cached_completion(query, ctx):
    key = sha256(normalize(query, ctx))

    # Tier 1: exact-key lookup
    if (hit := redis.get(key)): return hit

    # Tier 2: semantic search
    emb = embed(query)
    match = vector_db.search(emb, top_k=1, filter=scope_of(ctx))
    if match and same_scope(match, ctx) and match.score >= threshold_for(category(query)):
        remaining = match.expires_at - now()
        if remaining > 0: redis.set(key, match.response, ttl=remaining)
        return match.response

    # Miss on both tiers: call the model
    resp = llm(query, ctx)
    if is_valid(resp):
        ttl = ttl_for(category(query))
        redis.set(key, resp, ttl=ttl)
        vector_db.insert(emb, resp, ttl=ttl, scope=scope_of(ctx))
    return resp

Key Considerations for Cache Design

• Cache Key Granularity: Keys must encapsulate all factors that could alter an LLM's response, including query text, context documents, model ID/settings, retrieved source versions, and user access scope. This prevents incorrect cache hits due to varying inputs or permissions.
• Freshness and TTLs: Time-to-Live (TTL) values should be dynamic and depend on the data's staleness tolerance. For instance, live market data requires very short TTLs, while static HR policy documents can have longer ones. Invalidating cached entries upon source content updates is crucial.
• Similarity Threshold Tuning: The cosine similarity threshold for semantic caching is workload-dependent. Code-like queries might need higher thresholds (0.95+), while conversational queries can tolerate lower ones (0.85-0.90). This requires empirical tuning.
• Validation and Warming: Implement validation checks for LLM responses before caching them (e.g., no errors, valid JSON). Consider warming the cache with common queries and running the cache in shadow mode to evaluate its effectiveness and prevent poisoning with bad data.
• When to Skip Caching: Avoid caching for requests with personal data, creative tasks that require varied outputs, and genuinely real-time data where even minor staleness is unacceptable.

The principles of memoization and caching predate LLMs, emphasizing the timeless nature of optimizing redundant computations. Applying these established patterns to the unique billing and performance characteristics of LLM APIs is key to building cost-effective and responsive AI-powered applications.