Software Architecture and System Design News
Latest curated articles from top engineering blogs
Shopify
AWS133 articles
Labyrinth 1.1: Enhancing End-to-End Encrypted Message Backups
Labyrinth 1.1 is an update to Meta's end-to-end encrypted storage system for Messenger, focusing on improving the reliability of message backups. The new sub-protocol ensures messages are immediately placed into the recipient's encrypted backup, preventing data loss even if the sender's device goes offline or is lost. This architectural enhancement significantly boosts data durability for E2EE communications.
AWS Architecture Blog·2d agoReal-time Analytics Architecture with AWS for ERP Systems
This article details how Oldcastle APG migrated from on-premises ERP reporting to a real-time analytics solution on AWS, integrating Infor Cloud ERP with Amazon Aurora and Amazon QuickSight. It outlines the architectural patterns and specific AWS services used to overcome challenges like batch processing delays and limited reporting capabilities, enabling immediate operational insights and a seamless user experience.
LinkedIn's Unified Platform for AI-Driven Hiring Data
LinkedIn developed a unified integrations platform to consolidate fragmented hiring data pipelines, standardizing and reconciling data from various sources like applicant tracking systems and job boards. This platform improves data quality, speeds up partner onboarding, and provides a consistent, scalable foundation for downstream AI applications in talent systems. The architecture focuses on standardization, orchestration, and enhancement layers to process and deliver clean data.
AWS Aurora Serverless v4: Faster Scaling and Improved Throughput for Database Workloads
AWS has significantly enhanced Aurora Serverless with Platform Version 4, offering 45% faster ramp-up during demand spikes and 30% higher throughput. These improvements stem from smarter scaling algorithms and better resource scheduling, making Aurora Serverless a more compelling option for dynamic and bursty workloads that benefit from automatic capacity adjustments.
CAP Theorem Explained: Consistency, Availability, and Partition Tolerance in Distributed Systems
This article explains the fundamental CAP Theorem, which posits that a distributed system can only guarantee two out of Consistency, Availability, and Partition Tolerance at any given time. It clarifies that Partition Tolerance is unavoidable in distributed systems, forcing the real design choice between Consistency (CP) and Availability (AP) during network partitions. Using MySQL master-slave replication as an example, the article demonstrates how replication lag illustrates the AP tradeoff, where availability is prioritized over strong consistency.
Modernizing Infrastructure for AI Success: Overcoming Legacy Tech Debt
This article, based on an IDC survey, highlights the critical connection between modern underlying technology stacks and successful AI outcomes. It reveals that organizations embracing continuous modernization strategies for their data and infrastructure experience significantly higher digital revenue growth and lower AI project failure rates compared to those burdened by technical and data debt, especially with legacy relational databases.
Database Investigator: AI-Driven Database Performance Diagnostics
This article introduces Datadog's Database Investigator, a tool designed to automatically identify and resolve database performance issues. It leverages an agentic approach to analyze query patterns, resource utilization, and system metrics, providing actionable insights for database administrators and developers to optimize database health and application performance.
Architectural Strategy for Migrating Legacy Database-Centric Systems with Event Sourcing
This article outlines an architectural strategy for migrating legacy database-centric systems using events and progressive ownership transfer. It focuses on how to incrementally modernize monolithic applications by extracting functionalities and data, leveraging event-driven patterns to decouple services and manage data consistency during the transition.
Optimizing Hadoop Big Data Workloads on Arm-based AmpereOne Processors
This article explores the setup, tuning, and performance evaluation of Hadoop on AmpereOne Arm-based processors, highlighting their power efficiency and cost advantages for big data workloads. It delves into the architectural benefits of AmpereOne processors, Hadoop's compatibility with Arm, and provides practical guidance for deploying and optimizing Hadoop clusters on this infrastructure. The focus is on leveraging modern hardware for scalable and cost-effective big data processing.
Replacing Database Sequences at Scale: A Distributed ID Generation System
This article details Coupang's journey to replace legacy database sequences with a highly available, low-latency distributed ID generation system without breaking over 100 existing services. The solution leverages local application caching, server-side caching, and DynamoDB as the source of truth, optimizing for performance and availability over strict global ordering and gap-free IDs. It highlights practical design principles for large-scale migrations, emphasizing simplicity and backward compatibility.
Replacing Database Sequences at Scale: A Cached, Distributed ID Generation System
This article details Coupang's journey to replace traditional database sequences with a highly scalable, available, and low-latency distributed ID generation system. It highlights critical design decisions, such as prioritizing eventual consistency and local caching over strict global ordering and network calls, to support over 100 services and facilitate a seamless migration from relational databases to NoSQL.
Scaling Challenges with Misused Vector Databases
This article highlights a common architectural pitfall where a system broke during scaling not due to performance bottlenecks, but incorrect database selection. The author mistakenly used a vector database for both similarity search and general data storage, leading to poor performance and scalability issues. The solution involved adopting a hybrid architecture, leveraging a vector database for its strengths (semantic search) and a traditional database for its (exact-match queries and structured data storage).