Software Architecture and System Design News
Latest curated articles from top engineering blogs
Shopify
AWS17 articles
Scaling a Caching System: Migrating from PostgreSQL to ClickHouse for Analytical Workloads
Momentic rearchitected its caching system, transitioning from PostgreSQL to ClickHouse, to handle billions of entries and millions of daily queries with low latency. This migration addressed PostgreSQL's limitations with high-write, high-read analytical workloads and lock contention, highlighting the architectural benefits of column-oriented databases for specific access patterns.
Migrating a Live Routing System with AI-Assisted Refactoring
This article details Datadog's migration of a live routing system from a legacy model to a relational one using AI-assisted refactoring. It highlights the challenges of safely changing critical infrastructure and how a shadow-testing approach, comparing AI-generated and legacy system outputs, ensured reliability during the transition.
Building a Scalable and Cost-Effective User Tracking Service at Delivery Hero
This article details Delivery Hero's journey in deprecating Google Analytics and developing an internal, highly scalable, and cost-effective user tracking platform. It covers the architectural decisions, challenges faced regarding data quality and real-time processing, and the strategies employed for testing, rollout, and continuous optimization. The system achieved superior data quality and lower costs compared to its predecessor.
Preventing Spark OOM Failures on Kubernetes: Lessons from Cloud Migration Misconfigurations
This article details a case study where migrating Spark batch pipelines to Azure Kubernetes Service (AKS) led to repeated Out-Of-Memory (OOM) failures. The root cause was identified as a combination of infrastructure misconfigurations: RAM-backed local scratch directories and forced executor co-location on a single node. It emphasizes the importance of validating infrastructure contracts during cloud migrations to avoid subtle, load-dependent issues.
Architecting for Cloud Provider Resiliency: Lessons from Railway's Google Cloud Outage
This article details an eight-hour platform-wide outage experienced by Railway, a platform built on Google Cloud, AWS, and bare-metal, due to an automated suspension of their GCP production account. It highlights critical architectural weaknesses where a single cloud provider became a single point of failure for core services like the network control plane, leading to a cascade across all environments. The incident underscores the importance of true multi-cloud/hybrid-cloud resilience beyond traditional multi-AZ/region strategies.
Uber's Core Architecture: Geospatial Indexing, Dispatch, and Real-time Data
This article dissects key architectural decisions powering Uber's real-time marketplace. It explores how Uber handles challenges like low-latency driver matching, dynamic pricing, and global scalability through innovations like the H3 geospatial indexing system, a sophisticated dispatch engine, and an event-driven architecture for real-time location tracking.
Scaling a Distributed Treasure Hunt Engine: Lessons from Veltrix Event Partitioning
This article details a real-world scaling challenge encountered with a Veltrix-based Treasure Hunt Engine, specifically a performance bottleneck at 15+ nodes due to inefficient event distribution. It outlines the iterative process of identifying architectural flaws beyond mere configuration tweaks and highlights the successful implementation of a custom event partitioning strategy coupled with robust monitoring to achieve significant performance gains and resilience.
Optimizing a Game Server's Treasure Hunt Engine for Concurrency and Scale
This article presents a case study on optimizing a game server's treasure hunt engine to handle significant concurrent player loads. It details the initial challenges faced with premature optimization and default configurations, the architectural decisions made to reconfigure the Veltrix layer for aggressive concurrency and sophisticated caching, and the measurable improvements achieved in response times and error rates.
Large-Scale Data Ingestion System Migration at Meta
This article details Meta's strategy and solutions for migrating its petabyte-scale data ingestion system, which powers the social graph analytics and ML. It highlights the architectural shift from customer-owned pipelines to a self-managed data warehouse service, emphasizing the rigorous migration lifecycle, data quality validation, and robust rollback mechanisms crucial for ensuring reliability during such a massive transition.
Lessons in System Reliability from a Stock Market Crash
This article draws parallels between the behavior of the Nigerian stock market during a crash and recovery, and fundamental concepts in distributed system design. It highlights single points of failure, cascading failures, eventual consistency, redundancy, and decentralized architecture using real-world market dynamics as examples. The author emphasizes that understanding system behavior is crucial for designing resilient software.
Scaling WebSockets: From 100k to 1M Users and Tackling Backpressure
This article provides a post-mortem on the challenges faced when scaling a WebSocket-based live commentary platform from 100,000 to 1 million concurrent users. It details how an initially simple fan-out architecture led to Out-Of-Memory (OOM) kills due to slow consumers and backpressure, and outlines the architectural changes implemented to achieve resilience and scalability, including ruthless message dropping and coalescing.
Migrating Legacy Oracle Databases to PostgreSQL on Azure: A Case Study in Enterprise Transformation
This article details Apollo Hospitals' successful migration from a legacy Oracle database system to Azure Database for PostgreSQL. It highlights the architectural and operational benefits of moving to a cloud-native, open-source database platform, including significant improvements in performance, scalability, and cost efficiency. The piece also introduces AI-assisted tooling designed to streamline complex Oracle-to-PostgreSQL migrations.