Software Architecture and System Design News
Latest curated articles from top engineering blogs
Shopify
AWS389 articles
Scaling Cloudflare D1: Architecture for Millions of Serverless SQLite Databases
This article delves into the system architecture behind Cloudflare D1, a managed serverless SQLite database, focusing on how its control plane is designed to scale to support millions of databases. It highlights the architectural shift from traditional large database clusters to a model of many small, user-specific databases, detailing the principles and components that enable this massive scalability while maintaining performance at the edge.
Achieving Zero RPO and Sub-Second RTO with Active-Active Distributed Sagas
This article explores the architectural shift from active-passive to multi-region active-active topologies for mission-critical banking systems, focusing on distributed sagas and SQL databases. It details how to achieve zero RPO and sub-second RTO by leveraging synchronous consensus and resilient orchestrators like Temporal, addressing the mathematical trade-offs between latency and availability.
Lakehouse Data Layout Strategies: Benchmarking Liquid Clustering, ZOrder, and Partitioning
This article benchmarks different data layout strategies for Delta Lake lakehouse analytical systems: Partitioning, ZOrdering, and Liquid Clustering. It highlights that no single strategy is universally optimal, and the best choice depends on specific query patterns, impacting both cost and latency. The analysis provides practical recommendations and a decision framework for optimizing data access.
Designing a Foundation for Enterprise SQL Server ERP Systems
This article introduces ApexLedger ERP, a pre-built SQL Server database foundation aimed at standardizing common ERP functionalities like inventory, sales, general ledger, and role-based access control. It highlights the structured approach to database design, including tables, indexes, DML triggers for real-time stock management, and audit logging, which are critical elements in enterprise system architecture.
Optimizing Google Maps API Costs with a Two-Layer Caching Strategy
This article details a banking app's engineering decision to reduce Google Maps API costs and improve latency by implementing a two-layer caching strategy. The solution involves an internal database for persistent storage of map data and Redis for fast lookups of frequently accessed, fresh information, carefully balancing cost reduction with data freshness and compliance.
AWS Architecture Blog·3d agoArchitecting Highly Available Oracle Databases on AWS with EVS and FSx for ONTAP
This article outlines a robust architecture for deploying highly available Oracle databases on AWS using Amazon Elastic VMware Service (EVS) and Amazon FSx for NetApp ONTAP. It details how to achieve sub-millisecond storage latency, cross-region disaster recovery, and integrate with existing VMware operational workflows, focusing on storage, compute, and networking considerations. The design emphasizes a hybrid storage approach using vSAN for ephemeral data and FSx for ONTAP for persistent, replicated database volumes.
Database Internals for Portfolio Recommendation Apps
This article delves into the system design of a portfolio recommendation application, specifically focusing on how a single database can serve multiple roles: operational store, data lake, and data warehouse. It explores the architectural implications and internal mechanisms required to achieve this multi-purpose database utilization.
AWS Architecture Blog·3d agoDeploying Highly Available Oracle Databases on AWS with FSx for ONTAP
This article details the step-by-step deployment of Oracle Database on Amazon Elastic VMware Service (EVS) using Amazon FSx for NetApp ONTAP as NFS datastore storage. It focuses on practical implementation procedures for provisioning VMs, configuring storage, installing Oracle 19c, and setting up SnapMirror for cross-region disaster recovery. While the article is highly procedural, it touches upon underlying architectural decisions, such as storage sizing and high availability considerations, by referencing a related architectural planning post.
Designing Reliable Job Retries with Queues vs. Database Polling
This article explores architectural decisions for handling failed jobs, specifically reservation expiry in a healthtech context. It compares using a message queue with a Dead-Letter Queue (DLQ) and redrive mechanisms against a manual database polling approach. The core trade-off revolves around latency requirements, operational costs, and the complexity of ensuring idempotency, backoff, and poison-message handling.
Database Workload Capture and Replay System for Scalability and Compatibility
Airbnb developed a robust system to capture and replay real production database workloads. This system addresses critical challenges like capacity planning, de-risking database upgrades, and performance debugging by simulating production traffic in test environments. By leveraging ProxySQL as a client-agnostic capture point, the architecture ensures accurate transaction replay and result comparison across different database versions or configurations.
Pinterest's Metrics Board: An End-to-End Metrics Platform
Pinterest built Metrics Board, an end-to-end platform to automate metric creation, enforce quality, and enable agent-driven analytics for their petabyte-scale data lake. This system design centralizes metric definitions, automates computation via Airflow DAGs, ensures data quality, and integrates with various downstream products like data catalogs and BI tools.
Distributed Databases and Storage: Insights from Cockroach Labs CTO Peter Mattis
This article is an interview with Peter Mattis, co-founder and CTO of Cockroach Labs, discussing his journey from Google (Gmail, Colossus) to building CockroachDB. It offers deep insights into the architectural decisions, trade-offs, and underlying data structures like B-trees and LSM-trees crucial for building scalable, reliable, and consistent distributed databases.