Software Architecture and System Design News
Latest curated articles from top engineering blogs
Shopify
AWS254 articles
AWS Architecture Blog·5h agoMapfre USA's Fraud Detection Platform on AWS: A Lakehouse and ML Approach
Mapfre USA modernized its fraud detection capabilities by implementing a scalable platform on AWS, integrating machine learning models with graph-based features to identify complex fraud patterns. The architecture leverages a lakehouse design with Apache Iceberg, Amazon EMR Serverless, and Neo4j for data processing and enrichment. This system aims to improve fraud detection accuracy and claims handling efficiency by automating alerts and providing explanations to adjusters through Guidewire integration.
AWS Architecture Blog·1d agoScalable Petabyte-Scale Video Storage and Analytics on AWS
This article details March Networks' architecture for cloud-based enterprise video surveillance storage and analytics on AWS. It highlights the transition from on-premise NVRs to a hybrid cloud model using Amazon S3 and S3 Glacier for cost-effective, petabyte-scale video retention and lifecycle management. The architecture supports secure ingestion, monitoring, and advanced analytics, addressing the challenges of distributed video data management.
Datadog's Test-Driven Migration from KV to Relational Database for Stream Routing
Datadog engineers migrated a critical production system, Stream Router, from an eventually consistent key-value store to PostgreSQL to overcome transaction size limits and improve performance. This migration involved a careful schema redesign and a test-driven refactoring process, significantly accelerated by AI tools like Claude and Cursor. Key architectural decisions included modularity, a comprehensive test suite, and a blue/green deployment strategy for a smooth transition.
Designing a Web Search Engine: Inverted Index, Scatter-Gather, and Two-Phase Ranking
This article explores the core architectural principles behind web search engines like Google, focusing on how they efficiently retrieve and rank billions of documents. It details the use of an inverted index for fast keyword-to-document mapping, the scatter-gather pattern for distributing queries across sharded indices, and a two-phase ranking approach to balance performance with relevance.
E-commerce Mobile App Architecture with Flutter and Scalable Backend
This article discusses the architectural approach for building a cross-platform e-commerce mobile application using Flutter for the frontend and a decoupled backend. It highlights the importance of separating concerns, leveraging a REST API, and optimizing performance through various techniques to achieve scalability and a smooth user experience.
Batch vs. Streaming Data Processing Architectures
This article explores the fundamental architectural differences between batch and streaming data processing, focusing on the trade-offs between data completeness and processing latency. It discusses various strategies, windowing techniques, and architectural patterns like Lambda and Kappa, which are crucial for designing robust data pipelines.
AWS Architecture Blog·5d agoSpecification-Driven Composition for Flexible Data Workflows
This article introduces specification-driven composition as an architectural pattern to build scalable and flexible data pipelines. It addresses common issues in script-based pipelines like logic duplication and governance challenges by separating workflow intent from implementation. The pattern utilizes specifications to declaratively define data transformations, enabling dynamic pipeline assembly from reusable components and improving operational consistency.
Bloom Filters for Fast Membership Checks in Large-Scale Systems
This article explains Bloom filters, a probabilistic data structure used for efficient membership testing in large datasets. It delves into their underlying principles, construction, and operational characteristics, highlighting their utility in system design for reducing expensive disk or network operations at the cost of potential false positives.
Cloudflare's Meerkat: Global Consensus with QuePaxa for Control Plane State
Cloudflare introduces Meerkat, an experimental distributed consensus service designed to manage control-plane state across its global network of 330+ data centers. Unlike traditional consensus algorithms like Raft, Meerkat leverages QuePaxa, which allows all replicas to perform writes and avoids timeouts, making it robust in unpredictable wide-area network conditions. This architecture aims to provide linearizable consistency and strong fault tolerance for critical services.
Cache Invalidation Strategies for Distributed Systems
Cache invalidation is a fundamental challenge in distributed systems, addressing how to ensure data consistency between caches and the source of truth. This article explores common strategies for maintaining data freshness and discusses the trade-offs involved in choosing an appropriate invalidation approach.
Achieving Offline-First Durability for Web Applications with IndexedDB
This article explores architectural patterns for building robust offline-first Progressive Web Applications (PWAs), focusing on client-side data durability. It demonstrates how to persist both application state (user documents) and in-flight operations (AI jobs) using IndexedDB, ensuring user work is never lost due to browser refreshes or network interruptions. The core principle involves writing job intent to local storage *before* network requests to guarantee recovery.
AWS Architecture Blog·7d agoMulti-Region SQL Server Disaster Recovery with AWS FSx for NetApp ONTAP
S&P Global implemented an innovative multi-region disaster recovery solution for their Capital IQ platform using Amazon FSx for NetApp ONTAP. This strategy leverages SnapMirror replication for cross-region data consistency and FlexClone technology for rapid, sub-15-minute read-only failover, followed by a full read-write recovery process. The solution addresses critical RTO/RPO requirements for financial data while optimizing costs.