Airbnb's Sitar-Agent: Dynamic Configuration Sidecar for Kubernetes
This article details the architecture of Airbnb's Sitar-agent, a Kubernetes sidecar designed for dynamic configuration distribution across a large microservices fleet. It highlights key architectural decisions such as using a sidecar over libraries, snapshot-based bootstrapping from S3 for resilience, and migrating from Sparkey to SQLite for local storage. The system focuses on ensuring high availability of configuration data and rapid propagation of updates while minimizing dependencies on centralized infrastructure.
Read original on InfoQ ArchitectureAirbnb's Sitar-agent is a critical component in their microservices infrastructure, responsible for delivering dynamic configuration updates to tens of thousands of pods. It operates as a Kubernetes sidecar, abstracting configuration delivery logic from application services and ensuring language independence across various programming languages like Java, Python, Go, TypeScript, and Ruby. The system is engineered for high availability and rapid propagation of changes, enabling engineers to modify application behavior without service redeployments.
Architectural Overview and Key Decisions
The Sitar-agent employs a pull-based model, polling a backend service for updates every ten seconds. This approach, combined with server-side caching and incremental change tracking, minimizes backend load while delivering updates within tens of seconds. A core architectural decision was the use of a sidecar pattern over embedding configuration logic directly into application libraries. While a library-based approach might reduce resource consumption, the sidecar simplifies operations, reduces implementation effort across languages, and ensures consistent behavior.
Sidecar Pattern Benefits
The sidecar pattern centralizes cross-cutting concerns (like configuration management, logging, metrics collection) into a separate container. This decouples the concern from the application logic, allowing for language independence, easier updates, and consistent behavior across heterogeneous services. It's often chosen over library-based solutions when complexity, operational overhead, and maintenance across multiple languages become significant challenges.
Resilience and Performance Enhancements
- Snapshot-based Bootstrapping: To improve startup performance and operational resilience, Sitar-agent retrieves the latest configuration snapshot from Amazon S3 upon pod startup. This minimizes dependency on the central configuration service and allows applications to start even if the backend is temporarily unavailable.
- Local Datastore Migration: The team migrated from Sparkey to SQLite for the local datastore. SQLite was chosen over RocksDB due to its concurrency model, operational simplicity, and broad language ecosystem support. This decision was carefully managed with shadow-read validation and feature-flag-controlled rollouts to mitigate migration risks.
- Java Rewrite: The modernization involved a rewrite in Java, contributing to improved reliability and overall system performance.
These improvements collectively enhance the system's ability to provide configuration data reliably and quickly, even in the face of service disruptions or high update frequency. The design prioritizes decentralization of access to configuration data through the local sidecar, reducing the blast radius of central service failures.