Slack's Shipyard: Building an Immutable EC2 Platform for Modern Infrastructure
Slack's Shipyard platform modernizes EC2 instance management by shifting from mutable, long-lived instances to an immutable, artifact-driven approach. It integrates image baking, progressive rollouts, and automated safety checks, treating infrastructure as deployable artifacts rather than continuously updated servers. This architectural shift enhances reliability, security, and predictability for large-scale EC2 deployments, especially for workloads not suitable for containers.
Read original on Slack EngineeringSlack's journey to modernize its EC2 infrastructure culminated in Shipyard, a platform designed to bring modern deployment practices like immutability, progressive rollouts, and automated safety directly to EC2 instances. This represents a fundamental shift from their previous model of continuously updating long-lived instances using Chef, which was hitting its limits in terms of service-level deployments, infrastructure drift, and coordination complexity.
Core Principles of Shipyard
Shipyard treats infrastructure as deployable artifacts. Instead of relying on continuous configuration management on running instances, configuration is primarily applied during image baking and initial provisioning. This approach minimizes configuration drift and makes system behavior more predictable. Key capabilities include multi-architecture/OS support, metrics-driven deployments with automated safety checks, and fast, predictable provisioning.
Immutable Infrastructure with Golden Images
Immutability vs. Mutability
Mutable infrastructure allows in-place changes, leading to configuration drift and 'snowflake' servers. Immutable infrastructure, as implemented in Shipyard, means instances are never changed after deployment; any update requires replacing the instance with a new one built from a fresh image. This significantly improves consistency and reliability.
- slack-zero Base Image: A shared, standardized "golden" base image (like a Docker base image) built by the Compute Platform Team using AWS Image Builder. It includes OS baseline, networking, service discovery, monitoring, and security agents.
- Service Images: Individual service teams build their own AMIs on top of `slack-zero`, defining software installations and service configurations. This layered approach combines platform stability with team-level flexibility.
- Image Baking and Provisioning: Heavy operations (package installation) occur during the 'bake' phase into the AMI, while environment-specific settings (secrets, regional config) are applied during a lightweight 'provisioning' phase at boot time. This ensures fast instance startup.
Deployment and Fleet Management
Deployments involve building a new AMI and rolling it out by replacing existing instances, rather than patching them. Slack uses AWS Instance Refresh for Auto Scaling Groups and Karpenter for Kubernetes worker fleets. Their global orchestrator, Gondola, supports progressive rollouts with metric-based automated safety checks and rollbacks. An inventory system, Peekaboo (built with AWS EventBridge, OpenSearch, and Lambda), provides real-time visibility into the EC2 fleet.
Emergency Workflows
While immutability is the goal, Shipyard acknowledges the need for emergency fixes. It allows targeted configuration changes on running instances via AWS Systems Manager and predefined Chef recipes, but with the expectation that affected instances will be replaced via regular deployment pipelines afterward to restore the immutable state.