Platform Engineering Best Practices at GitHub
This article from GitHub Engineering outlines the shift to platform engineering and discusses essential practices for tackling platform problems. It emphasizes understanding the domain, mastering core infrastructure skills like networking and distributed systems, and the importance of knowledge sharing within a platform team. The article also highlights the wider impact radius of platform changes and the unique challenges in testing distributed infrastructure.
Read original on GitHub EngineeringPlatform engineering focuses on building the foundational tools and services that product engineers utilize to create end-user products. Unlike product engineering, which directly addresses external customer problems, platform engineering serves internal customers, providing the infrastructure, automation, and core services necessary for reliable and scalable product development. This shift requires a distinct set of skills and problem-solving approaches.
Core Skills for Platform Engineers
Platform teams require a deeper understanding of underlying technical domains due to their role as the foundational layer. Critical areas include:
- Network Fundamentals: A strong grasp of TCP, UDP, L4 load balancing, and debugging tools (e.g., dig) is essential to understand traffic impact.
- Operating Systems and Hardware: Knowledge of VMs, physical hardware selection, and OS choices is crucial for scalability, cost management, and security.
- Infrastructure as Code (IaC): Proficiency with tools like Terraform, Ansible, and Consul automates infrastructure provisioning and reduces human error.
- Distributed Systems: Acknowledging the inevitability of failures and implementing proactive solutions like failover and recovery mechanisms are vital for reliability.
Impact Radius and Testing in Distributed Systems
Wider Impact of Platform Changes
Even minor changes to foundational services, like DNS, can have extensive repercussions across numerous dependent products. Understanding downstream dependencies and employing robust monitoring are critical for managing this risk.
Testing changes in distributed environments, especially for core services, presents significant challenges. Strategies include using dedicated test sites, thorough IaC testing (provisioning/deprovisioning), and End-to-End (E2E) testing with partial traffic redirection. Implementing self-healing capabilities helps identify bottlenecks proactively, and a host-by-host rollout strategy allows for individual machine rollback to minimize impact.