Automating Code Review with AI: Duolingo's PR Risk Assessment System
This article discusses Duolingo's approach to integrating AI into engineering workflows, specifically focusing on their automated code review system. It highlights the cultural and educational initiatives undertaken to build engineer trust in AI, culminating in a PR Risk Assessment bot that auto-approves low-risk changes. The system design involves classifying pull requests by risk level to streamline the development process and accelerate delivery without compromising quality.
Read original on InfoQ ArchitectureThe Challenge of AI Adoption in Engineering Workflows
Integrating AI into established engineering processes like code review presents significant challenges beyond just providing tooling access. Engineers can be skeptical of AI's capabilities, especially when core systems are perceived as 'sacred' and have been honed over years. Furthermore, AI changes the dynamics of accountability and trust within a team, as responsibility for code quality shifts when an AI agent is involved. Duolingo addressed these challenges by focusing on cultural change, AI literacy, and building autonomous systems with clear guardrails.
Building AI Literacy and Trust Through Education
Duolingo's strategy for successful AI adoption centered on comprehensive education and support. This included structured, hands-on workshops, live office hours for AI support, and fostering shared learning through internal communities. These initiatives aimed to increase engineers' understanding of AI's strengths and weaknesses, reducing fear and uncertainty, and ultimately preparing them for adopting AI agents in critical workflows like code review. Observability dashboards tracking AI tool usage and cost were also crucial for understanding adoption patterns and communicating value to leadership.
Case Study: Redesigning Code Review with a PR Risk Assessment Bot
The core of Duolingo's autonomous code review system is a PR Risk Assessment bot. This agent categorizes code changes into 'low,' 'medium,' or 'high' risk, or 'undetermined' if the change is too large (exceeds token limits). For changes classified as 'low-risk' and meeting other predefined criteria, the system auto-approves the pull request, allowing it to merge without human intervention. This system aims to alleviate bottlenecks caused by human code review, especially given the increased speed of code generation by AI, while maintaining or improving quality.
Key System Design Principle
When designing an autonomous system for critical workflows, prioritize mechanisms that build trust and provide safety nets. Duolingo's approach of auto-approving *only* low-risk changes, combined with a robust education program, exemplifies this principle, mitigating the 'fast to build, fast to lose' nature of trust in AI.
Architectural Considerations for the PR Risk Assessment System
- Risk Classification Logic: The system requires sophisticated logic to accurately determine the risk level of a pull request. This likely involves analyzing code diffs, testing outcomes, author history, and possibly external dependencies.
- Integration with SCM and CI/CD: Seamless integration with the version control system (e.g., GitHub) and the CI/CD pipeline is essential for the bot to monitor PRs, trigger risk assessments, and perform auto-approvals.
- Token Limit Handling: The 'undetermined' category for large PRs highlights a practical limitation and a design decision: rather than making potentially risky guesses, the system defers to human review when input exceeds an AI model's capacity.
- Guardrails and Criteria for Auto-Approval: Beyond 'low-risk,' additional criteria are necessary to prevent regressions. These might include passing all automated tests, specific file types, or changes within well-understood, low-impact areas of the codebase.
- Observability and Feedback: Monitoring the bot's performance, auto-approval rates, and any subsequent defects from auto-approved PRs is crucial for continuous improvement and maintaining confidence in the system.