Software Architecture and System Design News
Latest curated articles from top engineering blogs
Shopify
AWS1161 articles
Designing a Distributed Cache with Consistent Hashing and Virtual Nodes
This article explores the challenges of distributed caching, specifically focusing on achieving consistent data placement and minimizing key reshuffling when scaling a cache cluster. It proposes using consistent hashing with virtual nodes as a robust solution to address these issues, demonstrating its benefits over naive hash-modulo approaches.
Optimizing Generative AI for Recommendations at Scale: Spotify and NVIDIA TensorRT-LLM
This article discusses Spotify's collaboration with NVIDIA to optimize generative AI models for real-time recommendation inference using TensorRT-LLM. It focuses on the engineering challenges of deploying large language models (LLMs) for personalized experiences, highlighting the need for efficient inference to reduce latency and computational costs at Spotify's scale.
Netflix Conductor's Architectural Evolution for Scalable Workflow Orchestration
This article details the architectural evolution of Netflix Conductor, a workflow orchestration engine critical for stitching microservices into reliable business processes. It highlights how Netflix re-architected Conductor from the ground up to handle massive growth, addressing scaling challenges by modernizing its data plane, rewriting its evaluation engine, and introducing asynchronous processing. The evolution showcases key decisions in datastore migration, queueing systems, and task management to achieve high throughput and reliability.
AWS European Sovereign Cloud's Backbone Disconnection Test for Resilience
This article discusses AWS's test of its European Sovereign Cloud by temporarily disconnecting it from the global network backbone. The test aims to demonstrate the region's operational resilience and compliance with EU data sovereignty regulations, highlighting the architectural considerations for maintaining services independently. It explores the technical aspects of network isolation versus legal data sovereignty, posing questions about sustained autonomy for cloud regions.
Hyperdrive MySQL Caching Bug: A Deep Dive into Protocol Mismatches in Distributed Connection Pooling
This article details a subtle and rare caching bug in Cloudflare's Hyperdrive, a distributed database connection pooler, specifically impacting MySQL support. The bug originated from a mismatch in MySQL client/server protocol capabilities between Hyperdrive's internal nodes and the Workers' client drivers, leading to incorrect cache reads and empty query responses. The resolution involved canonicalizing cached query results to Hyperdrive's internal capability set.
Evolving Netflix's Ads Event Pipeline for Live Streaming with Apache Flink
This article details how Netflix re-architected its Ads Event Publisher, moving from a database lookup-based enrichment system to a stateful stream processing solution using Apache Flink. The evolution was driven by the demanding scale and spiky traffic patterns of live streaming, focusing on removing critical path dependencies, improving scalability, and ensuring event freshness. The new architecture leverages real-time streaming joins to match ad serving responses with client events, enhancing efficiency and resilience.
Optimizing AI Discovery: Cloudflare's Role in Enhancing Search Indexing for Large Language Models
This article details a joint research pilot between Cloudflare and OpenAI focused on improving AI content discovery. By leveraging Cloudflare's network signals, OpenAI's crawlers can more efficiently find and index valuable web pages, including previously unknown content. This collaboration aims to provide LLMs like ChatGPT with richer, more relevant, and up-to-date information, ultimately enhancing user experience and website discoverability.
Build vs. Buy for Metrics Dashboards: Architectural Decisions for SaaS Backends
This article explores the architectural trade-offs between building an in-house metrics dashboard backend using tools like PostgreSQL and Metabase versus integrating with a third-party metrics ingestion API for SaaS applications. It emphasizes distinguishing between raw event data for incident investigation and aggregate metrics for operational trends, guiding decisions based on business needs, team size, and the need for durable, traceable evidence.
Uber's Automated Dependency Analysis in a Service Mesh
Uber developed an automated system to map and classify service dependencies within its microservices architecture to identify how downstream failures propagate. This system uses RPC middleware to correlate caller and callee failures, classifying dependencies as fail-close, fail-open, or unknown, offering faster feedback than traditional distributed tracing for rare failures.
Monoliths vs. Distributed Systems: Trade-offs in Consistency, Latency, and Security
This article discusses the fundamental trade-offs between monolithic and distributed system architectures, particularly focusing on consistency, latency, availability, and security. It highlights how monoliths generally excel in performance and consistency due to fewer moving parts, while distributed systems offer superior security through concepts like least privileged access.
Scaling Live Sports Streaming to 32 Million Concurrent Viewers: Lessons from JioHotstar
This article details the engineering strategies and lessons learned by JioHotstar (now JioCinema) in successfully live-streaming the Indian Premier League (IPL) cricket finals to a record-breaking 32 million concurrent viewers. It highlights critical system design considerations for high-uncertainty events, focusing on capacity planning, identifying golden paths, rigorous testing methodologies, and managing latency budgets across distributed systems.
Netflix TimeSeries Tiered Storage: Kafka/Flink and Native Cassandra Cold Reads
Netflix's TimeSeries platform addresses the challenge of managing petabytes of temporal event data by implementing a tiered storage strategy. This involves moving older, less frequently accessed data from expensive hot Cassandra storage to cheaper S3 object storage while maintaining a consistent API and acceptable latency. The article details two primary solutions: an initial Kafka/Flink-based streaming approach and an evolved, more cost-effective native Cassandra cold tier leveraging S3 backups.