Software Architecture and System Design News
Latest curated articles from top engineering blogs
Shopify
AWS298 articles
Consistent Hashing for Distributed Caching and Databases
This article explains consistent hashing, a crucial algorithm enabling distributed systems like Cassandra, DynamoDB, and Discord to scale efficiently. It highlights the fundamental problem with naive hashing (modulo N) in dynamic environments and demonstrates how consistent hashing addresses rebalancing issues and minimizes data movement during node additions or removals.
Transactional Background Job Queues in Rust: Addressing Dual-Write and Idle CPU
This article introduces Azums, a Rust-native transactional background job engine designed to solve common distributed system challenges like the dual-write problem and inefficient idle CPU usage. It achieves this by integrating job enqueuing directly within database transactions and leveraging native database notification systems for zero-cost idle execution. The solution provides a unified API across various storage backends, offering a robust approach to asynchronous task processing in distributed architectures.
Designing Data Jobs for Resilience: Surviving Retries and Partial Failures
This article discusses five architectural patterns for building robust data processing jobs that can gracefully handle retries, partial failures, and duplicate executions. It emphasizes the importance of designing data pipelines to be idempotent and resilient to ensure data integrity and system reliability in distributed environments. The core focus is on ensuring that data jobs produce correct and consistent results even when faced with transient errors or unexpected restarts.
Architecting a High-Throughput Solana Data Pipeline
This article outlines the system design for a resilient, zero-data-loss Solana swap indexer, emphasizing the unique challenges of high-speed blockchain data ingestion. It details an architecture leveraging Rust for performance, NATS for message brokering and backpressure handling, and ClickHouse for analytics, all within a co-located VPC to minimize latency and manage cloud economics effectively. The design focuses on decoupling ingestion from database writes and optimizing RPC usage.
Automating Database Remediation with Graph Search and State Machines at Stripe
Stripe engineered a sophisticated system to automate database incident recovery, addressing the limitations of their previous hard-coded remediation logic. By modeling their global MongoDB infrastructure as a graph and employing graph search algorithms with state machines, they dynamically compute and execute remediation plans. This approach significantly reduced manual intervention and pager alerts, improving the reliability and operational efficiency of their database fleet.
Netflix's Real-Time Distributed Graph Querying with gRPC
This article, part three of a series, details how Netflix designed a high-performance serving layer for its Real-Time Distributed Graph (RDG). It focuses on the architectural choices and trade-offs made to enable efficient querying of a massive, constantly evolving graph with sub-100ms latency, handling diverse workloads from shallow-wide lookups to deep-narrow traversals.
Zero-Downtime Data Evolution: Migrating Production Databases Without Users Ever Knowing
This article discusses strategies for achieving zero-downtime database migrations, a critical aspect of maintaining high availability in production systems. It focuses on techniques that allow schema and data changes to be deployed without interrupting user services, which is essential for robust distributed systems and agile development.
Data Pruning and API Contract: The Silent Impact of Data Retention on System Behavior
This article highlights a critical system design issue where data retention policies, especially in systems relying on 'absence of data' for positive assertions, can silently alter an API's perceived contract. It details how pruning old, irrelevant data in a `gs_orgs` table, intended to improve database efficiency, inadvertently changed the meaning of an API's 'good standing' response, as the API could not differentiate between truly clear records and those pruned due to age. The core lesson is that data lifecycle management must be explicitly tied to API output contracts to prevent logical errors.
Optimizing Read and Write Paths in Distributed Systems
This article explores the fundamental differences and trade-offs between optimizing read and write operations in high-traffic applications. It delves into various strategies like indexing, caching, read replicas, and CQRS, highlighting how each technique impacts data consistency, staleness, and failure modes across the read and write paths. Understanding these dynamics is crucial for designing scalable and reliable distributed systems.
Applying Redis and Kafka in Production: Common Interview Questions & System Design Patterns
This article distills practical insights for using Redis and Kafka in production systems, framed as common interview questions. It moves beyond theoretical answers to explain real-world patterns for caching, event-driven architectures, and distributed system challenges, drawing examples from a point-of-sale (POS) and inventory microsystem. Key discussions include Redis's single-threaded performance, mitigation of caching issues like cache penetration and breakdown, and strategic decisions for when not to use Redis.
Scaling Reads with Database Read Replicas
This article discusses the limitations of caching for personalized data and introduces read replicas as a solution for scaling database read throughput. It explains how read replicas work, the concept of read/write splitting, and highlights the crucial trade-off of replication lag and eventual consistency, offering strategies to manage it.
Optimizing Data Layers for Low-Latency AI Workloads with Valkey
This presentation explores architectural patterns for achieving microsecond-level latency in data layers, crucial for modern AI feature stores. It critiques traditional proxy-based architectures, highlighting their hidden costs and performance limitations, and advocates for direct-access Valkey architectures to improve resilience, reduce latency, and cut infrastructure costs.