Automating GitHub Org Hygiene with Evergreen: A Distributed Workflow Design
This article details GitHub's "Evergreen" system, a GitHub App designed to automate repository access reviews and stale repo archival within a shared organization. It highlights key architectural decisions such as using native GitHub features for choreography, storing state in Git, and implementing robust error handling and rate limiting to ensure reliability and scalability.
Read original on GitHub EngineeringThe article introduces Evergreen, a GitHub App that addresses the challenge of managing external collaborator access and repository lifecycle in large, shared GitHub organizations. It tackles manual housekeeping tasks like quarterly access reviews and stale repository detection, which can become unmanageable as an organization scales. Evergreen's design emphasizes fail-secure principles (revoking access by default if reviews are unanswered) and data immutability (never deleting repositories, only archiving, and logging all changes for auditability).
Architectural Design: One Brain, Many Hands
Evergreen's architecture is surprisingly simple, leveraging existing GitHub features rather than building custom infrastructure. It consists of a small Go webhook server, but the core intelligence lies in its choreography with GitHub's native capabilities. This approach minimizes operational overhead and maximizes platform integration.
- Controller Repository: A dedicated repo acts as the system's 'clock and memory.'
- Scheduled GitHub Actions: Trigger `repository_dispatch` events quarterly to wake up the app.
- GitHub App & Webhooks: The Go application listens for webhooks (e.g., `issues.closed`) to react to user interactions.
- Issues & Custom Properties: Used for all user interaction (access review checklists) and configuration.
- Git as State Store: Repository status and review details are stored as JSON files within the controller repo, leveraging Git history for an audit trail.
Key Design Principles and Implementations
Leveraging Platform Primitives
A crucial takeaway is the effective use of existing platform primitives (Git issues, webhooks, custom properties) to achieve complex workflow automation, circumventing the need for bespoke UIs, job queues, or separate audit databases. This simplifies the system significantly.
To handle scale and ensure robustness, Evergreen implements several critical mechanisms:
- Failure Handling: Employs a "fail closed" strategy where partial API failures cause the review to remain pending and be retried, ensuring eventual consistency and preventing silent inconsistencies.
- Rate Limiting: Proactively avoids GitHub's secondary rate limits by processing repositories in batches, spacing out API calls, and sleeping between batches. This design choice prioritizes stability over aggressive processing.
- Observability: Emits structured logs to Splunk for monitoring and debugging, essential for identifying and resolving edge cases in a production environment with many users.