Optimizing AI Inference Costs in a Code Reviewer System
Cloudflare significantly reduced the cost and latency of its AI code reviewer by 68% through aggressive optimization. This involved strategically leveraging LLM prompt caching, reducing the number of API calls, and carefully selecting models and effort levels. The article details the architectural adjustments and operational insights gained from this optimization effort.
Read original on Cloudflare BlogThe Challenge: Spiraling AI Inference Costs
Cloudflare's AI code reviewer saw its average cost per review double, primarily due to models becoming more capable (leading to larger, more complex code changes needing 'full reviews') and the adoption of more advanced, albeit more expensive, LLMs like Anthropic's Claude Opus. Critically, token usage per review tripled, from 0.91 million to 2.99 million, indicating increased context processing despite a high cache hit rate.
Key Optimization Strategies
The optimization effort focused on two main areas: code changes within the reviewer system (accounting for 40% of savings) and experimenting with LLM models and their 'effort' levels (the remaining 60%).
- Leveraging LLM Prompt Caching: The primary cost driver was identified as cache writes (43%) and cache reads (34%) due to repeatedly sending the entire conversation as context in each LLM call. Anthropic's prefix caching mechanism was underutilized. The fix involved splitting the system prompt to allow common tool definitions and agent prompts to be cached as shared prefixes across all repositories, drastically increasing cache hit rates for initial LLM calls.
- Reducing LLM Calls and Context Size: Several micro-optimizations were implemented to reduce the number of LLM calls per review. This included removing unused tools, reading files in parallel, skipping irrelevant reviewers on re-reviews, and stopping the conversation after the review was posted. Sharing a common prefix among multiple sub-reviewers (code quality, security, performance) further reduced redundant token usage.
- Model Selection and Effort Tuning: Cloudflare actively experimented with different LLM models (e.g., Anthropic Sonnet 5 vs. 5.5, Opus 5 vs. 5.5) and adjusted 'effort' levels (medium vs. high). This involved A/B testing models and effort levels to find the optimal balance between cost, performance, and review quality. For instance, moving to Sonnet 5.5 resulted in significant cost savings without a meaningful reduction in critical findings, primarily by using fewer tokens rather than a cheaper per-token price.
System Design Takeaway
When integrating with large language models, treat API calls and token usage as critical performance and cost metrics. Design your system to maximize caching opportunities for common prompts and context, minimize redundant information sent in each request, and strategically select models and inference parameters based on a clear understanding of cost-benefit trade-offs for different workloads.