Software Architecture and System Design News
Latest curated articles from top engineering blogs
Shopify
AWS729 articles
Robust LLM JSON Pipelines: The Three-Layer Validation Pattern
This article addresses a common challenge in AI/ML infrastructure: the unreliability of LLM-generated JSON outputs in production. It proposes a "Three-Layer Validation Pattern" to ensure robust, crash-free pipelines by moving beyond fragile regex parsing to schema-enforced validation and self-healing repair mechanisms. The architectural approach focuses on enhancing the reliability and resilience of systems integrating probabilistic LLM outputs.
Externalizing Authority for AI Agents in Software Development
This article discusses the challenges of using AI coding agents for larger software projects, particularly the tendency for agents to misinterpret instructions and drift from architectural plans. It proposes an architectural solution, arka.norn, which externalizes project authority and state governance from the AI model to a separate system. This approach aims to enforce architectural integrity, ensure accurate discovery, and provide reliable orchestration, thereby enabling more robust AI-assisted software development.
Optimizing LLM Inference with Speculative Decoding
This article explores speculative decoding, a technique to significantly speed up Large Language Model (LLM) inference by converting underutilized GPU capacity into output. It details how a smaller draft model generates candidate tokens, which are then verified in parallel by the larger target model, reducing sequential forward passes and maintaining output quality. This method addresses the memory bandwidth bottleneck in autoregressive decoding.
Scaling Pinterest's Home Feed Conditional Learned Retrieval System
This article details Pinterest's architectural evolution of its Conditional Learned Retrieval (CLR) system for the home feed, focusing on how they scaled its retrieval capabilities across diverse use cases and improved model foundations. It covers critical system design aspects such as efficient serving infrastructure, large-scale embedding management, and the unification of condition types to reduce engineering overhead.
Optimizing AI Usage for Cost Efficiency in Large-Scale Systems
This article details Datadog's strategies for managing AI inference costs, crucial for large-scale systems integrating AI. It focuses on agent evaluations, spending guardrails, and context optimization to balance performance and expenditure, offering insights into architectural decisions for sustainable AI deployment.
AWS Architecture Blog·14h agoGallup AI: Scaling Real-time Coaching with a Serverless RAG Architecture on AWS Bedrock
Gallup built an AI assistant, Gallup AI, to provide personalized leadership coaching by leveraging 90 years of proprietary workplace research. This system design focuses on a serverless architecture utilizing Amazon Bedrock for generative AI, RAG through Knowledge Bases and Kendra, and various AWS services for scalability, real-time responses, and data management. It demonstrates how to integrate large language models (LLMs) with private data for domain-specific AI applications.
Ramp's Internal AI Coding Agent: Architecture and Design of 'Inspect'
This article details Ramp's decision and architectural choices behind building 'Inspect', an in-house AI coding agent. It explores why a custom solution was preferred over third-party tools, focusing on the system's ability to provide remote sandboxed development environments with deep internal integrations for verification and autonomous debugging. The design emphasizes extensibility, allowing engineers to build hundreds of specialized agents on top of the Inspect platform.
Architecting Golden Paths for AI Agents in System Design
This article explores the architectural considerations for creating "Golden Paths" for AI agents, focusing on how system design principles must adapt when the primary users are autonomous agents rather than humans. It emphasizes the need for well-defined execution patterns, machine-consumable contracts, and robust dispatch controls to ensure agents operate reliably and efficiently within a larger system.
Agentic Context Management for AI Systems
This paper introduces Agentic Context Management (ACM) as an architectural discipline to address the critical memory and cost challenges in production AI agents. It proposes that managing agent context, including conversation histories, prompts, and tool outputs, is a lifecycle problem rather than just a storage-and-retrieval issue. The article outlines five primitives for ACM: architecting, ingesting, scoping, anticipating, and compacting & consolidation, making a strong economic case for validated compaction to achieve linear cost scaling with high fidelity.
Building Resilient LLM Integrations with Fallback Mechanisms
This article outlines a robust architectural approach to integrate Large Language Models (LLMs) into production systems, focusing on resilience against rate limits and transient outages. It advocates for a multi-layered defensive strategy involving exponential backoff, dynamic fallback routing, and graceful degradation to prevent cascading failures and ensure service continuity.
AWS Architecture Blog·1d agoDesigning a PHI/PII Detection and De-identification System for DICOM Images on AWS
This article details Clario's solution for automatically detecting and de-identifying Protected Health Information (PHI) and Personally Identifiable Information (PII) within DICOM medical images. The architecture leverages AWS services like Amazon Bedrock and Amazon Textract for scalable and compliant processing, addressing critical data privacy and regulatory requirements in clinical trials. Key design decisions involve using EKS for long-running, memory-intensive workloads and RDS for audit trail consistency.
Understanding LLM Serving Engines and Message Brokers for Distributed Systems
This article provides a comparative overview of LLM serving engines like Ollama, vLLM, and SGLang, highlighting their architectural differences and optimal use cases for various scales and interaction patterns. It also contrasts Apache Kafka and RabbitMQ, emphasizing their distinct roles as a distributed log and a message broker, respectively, crucial for designing robust distributed systems.