Datadog's Test-Driven Migration from KV to Relational Database for Stream Routing
Datadog engineers migrated a critical production system, Stream Router, from an eventually consistent key-value store to PostgreSQL to overcome transaction size limits and improve performance. This migration involved a careful schema redesign and a test-driven refactoring process, significantly accelerated by AI tools like Claude and Cursor. Key architectural decisions included modularity, a comprehensive test suite, and a blue/green deployment strategy for a smooth transition.
Read original on InfoQ ArchitectureThe Challenge: Scaling an Eventually Consistent KV Store
Datadog's Stream Router, an API for managing routing in its metrics pipeline, initially used an eventually consistent key-value (KV) database. As the routing table grew, this architecture faced severe limitations. The KV model required application code to reconstruct relational data, effectively making the application act as a relational database in-process. This led to thousands of sequential round trips, causing demanding operations to slow dramatically, sometimes taking up to 45 minutes, due to hitting transaction size limits and inefficient data retrieval.
Architectural Redesign: Embracing Relational Structure
To address the scaling issues, Datadog engineers redesigned the schema to explicitly reflect relationships between domain entities. This allowed them to leverage the capabilities of a relational database, using foreign keys to enforce relationships that were previously managed by complex application logic. This move from a KV model to a relational model (PostgreSQL) was a fundamental architectural shift aimed at improving data integrity, query efficiency, and overall performance.
Key Enablers for Migration Success
- Strong Code Modularity: The existing Stream Router was designed with high modularity, allowing the new PostgreSQL-backed implementation to adhere to the same API without requiring changes in upstream services. This was crucial for minimizing impact and facilitating a phased migration.
- Comprehensive Test Suite: A robust and extensive test suite was paramount. It served as the primary validation mechanism for AI-generated code changes, providing immediate pass/fail criteria and building trust in the refactoring process.
- Parallel Infrastructure (Blue/Green Deployment): Datadog employed a blue/green deployment strategy, running both the old FoundationDB-backed and the new PostgreSQL-backed Stream Routers in parallel. A dedicated validator service continuously compared responses between the two, alerting the team to any divergences and ensuring consistency before full cutover via feature flags.
System Design Lesson: The Power of Modularity and Testing
This case study highlights that strong architectural principles like modularity and comprehensive testing are not just good practices, but critical enablers for significant system migrations and refactoring efforts, especially when introducing new technologies or approaches like AI-assisted development. A well-tested, modular system is inherently more adaptable to change.
Post-migration, operations that once took 45 minutes now completed in approximately one second. Latencies dropped from hundreds of milliseconds to mere milliseconds. Data storage requirements decreased by up to 40x, and database costs were reduced by 90% due to more efficient handling of relationships and optimized query execution in PostgreSQL and DuckDB. CPU and memory usage also saw significant reductions, demonstrating the profound impact of choosing the right data model and database for the workload.
AI's Role in Refactoring and Its Limitations
AI tools like Claude and Cursor were used to accelerate the code refactoring process. Engineers provided the AI with the old implementation, the new schema, and a failing test, allowing the models to generate initial code passes. While effective for repetitive refactoring, the AI struggled with niche optimizations like batching, `UNNEST` tricks, or common table expressions, often generating correct but suboptimal queries. Human engineers had to intervene to optimize these, after which the models could replicate the pattern.
This case study illustrates a common challenge with AI code generation: while it can automate boilerplate and straightforward transformations, deep architectural optimizations and understanding of system-specific performance nuances often still require human expertise. AI is a powerful assistant but not yet a complete replacement for a seasoned engineer's insight into complex system interactions and performance characteristics.