Software Architecture and System Design News
Latest curated articles from top engineering blogs
Shopify
AWS564 articles
ZGateway: A Proxy Layer for Meta's ZippyDB
This article details ZGateway, a proxy layer introduced in front of Meta's ZippyDB key-value store to manage a vast and diverse client population. It highlights how ZGateway centralizes critical functionalities like connection management, request batching, admission control, and traffic routing, which were previously fragmented across millions of client binaries. The implementation addresses scalability and reliability challenges arising from a direct-access model, transforming an unmanageable many-to-many connection mesh into a controlled, efficient two-hop architecture.
Azure Multicloud Interconnect for AWS: Enhancing Hybrid Cloud Networking
This article introduces Azure Multicloud Interconnect for AWS, a new networking service designed to simplify and optimize connectivity between Azure and AWS environments. It aims to reduce latency, improve bandwidth, and enhance security for hybrid cloud architectures, addressing critical challenges in multicloud deployments.
AWS Architecture Blog·2d agoHybrid Cloud Orchestration for On-Premises Infrastructure with AWS Serverless
This article details a hybrid cloud orchestration solution leveraging AWS serverless services (Lambda, Step Functions, DynamoDB, EventBridge) to manage distributed on-premises infrastructure, including bare-metal servers and EKS Anywhere clusters. It addresses challenges like inconsistency, manual bottlenecks, and fragmented visibility across geographically dispersed data centers by providing a centralized control plane in the cloud with distributed execution on-premises. The architecture emphasizes event-driven workflows and a centralized inventory management system for robust and scalable operations.
Optimizing CDN Cache Storage with Zstandard Compression in Pingora
Cloudflare explored using Zstandard compression within their Pingora-based CDN cache to significantly reduce storage costs and inter-data center bandwidth. This "Cache Transcoding" approach encodes eligible assets upon entry into the cache and decodes them only when served to clients, trading a minor CPU increase for substantial space and network efficiency. The architectural decision focuses on optimizing resource usage in a massively distributed storage product.
Securing IEC 61850 Industrial Control Systems: A Deep Dive into Protocol Architecture and Vulnerabilities
This article explores the security architecture of IEC 61850, an industrial communication standard, focusing on its distinct communication models: MMS, GOOSE, and Sampled Values. It highlights how each protocol's underlying transport and communication paradigms (client/server vs. direct Ethernet) introduce unique attack surfaces and security considerations, making traditional security assessments inadequate. The research dissects protocol semantics and configuration to uncover potential manipulation points.
Building a Low-Latency Edge-Native Access Control System for Mega-Conferences
This article details the architectural design of a high-performance event telemetry and RFID access control engine engineered for large-scale conferences. It focuses on solving challenges like cellular saturation, Wi-Fi packet loss, and high I/O latency by implementing an edge-native architecture with local authentication, distributed locking for resource allocation, and asynchronous telemetry streaming to achieve sub-20ms access verification and real-time spatial analytics.
DoorDash Flux: Scaling Engineering Task Automation with Cloud-Based Agents
DoorDash developed Flux, a cloud platform that offloads engineering agent workloads from developer laptops to isolated cloud environments. This platform automates over 130,000 engineering tasks monthly, addressing limitations of local execution like resource constraints, security risks, and lack of observability. Flux leverages microVMs for isolated execution and an in-house gateway for secure internal system access.
Eliminating Long-Lived Credentials with Workload Identity Federation in GCP
This article explores Workload Identity Federation (WIF) in Google Cloud Platform, a crucial security and operational improvement for machine-to-machine authentication. WIF replaces error-prone, long-lived service account keys with a trust-based mechanism, issuing short-lived access tokens to external workloads. It details the architecture of WIF, its three core components (pool, provider, binding), and how to implement it for CI/CD systems like GitHub Actions and Harness.
Cloudflare's AI Search: An End-to-End System for Custom Data Retrieval
Cloudflare AI Search offers an integrated, end-to-end search and retrieval service for AI agents and applications over custom data. It leverages various Cloudflare primitives like Workers AI, Vectorize, and R2 to automate the entire search pipeline, from data ingestion to query response. This system simplifies the creation of specialized search engines for organizations, focusing on discoverability and scalable access to information.
Cloudflare Workers Now Support Inbound TCP and gRPC
Cloudflare Workers have expanded their capabilities to accept inbound TCP connections, moving beyond their original HTTP-only limitation. This enhancement enables full-duplex communication and supports various TCP-based protocols, with gRPC being the first to leverage this new feature. The change significantly broadens the types of workloads that can be handled at the edge, offering new architectural possibilities for message brokers, database proxies, and custom binary protocols.
Managed vs. Self-Hosted PostgreSQL: A System Design Perspective
This article explores the architectural considerations and operational trade-offs between using a managed PostgreSQL service and self-hosting PostgreSQL. It highlights how choosing between these deployment models impacts scalability, reliability, security, and cost-effectiveness for system designers and operations teams, guiding decisions based on business needs and technical capabilities.
Simplifying Container Deployments and Operations with AWS ECS Express Mode
This article introduces AWS ECS Express Mode, a new feature designed to streamline the deployment and management of containerized applications on AWS. It addresses the complexity often associated with setting up supporting infrastructure like load balancers, networking, and scaling policies, by offering a simplified, opinionated approach that still provides underlying control and extensibility for evolving workloads.