Harness Engineering: Building AI-Driven Software with Quality and Control
This article introduces "harness engineering" as a critical approach for developing AI-driven software, emphasizing human oversight "on the loop" rather than "in the loop." It advocates for extending continuous delivery principles to AI agents to ensure production-ready code, mitigate cognitive debt, and maintain quality and control in accelerated development environments.
Read original on The New StackThe Shift from Human "in" the Loop to Human "on" the Loop
As AI agents increasingly contribute to software development, the traditional "human in the loop" model, where humans directly review or intervene in every AI-generated output, becomes unsustainable and inefficient. Kief Morris of Thoughtworks proposes a shift to "human on the loop" through harness engineering. This approach focuses on designing the surrounding processes and guardrails that guide AI agents to produce high-quality, production-ready software, rather than micromanaging their output. It emphasizes defining "good" code and systems upfront and embedding these definitions into automated workflows.
Extending Continuous Delivery to AI-Driven Workflows
The core of harness engineering lies in adapting established continuous delivery (CD) principles to AI-accelerated development. Just as CD pipelines prevent bugs from reaching production by enforcing checks and tests, harness engineering aims to build similar automated validations for AI-generated code. This means: Fixing at the source: If an AI agent produces undesirable code, the solution isn't to manually correct the output, but to modify the "harness" (the instructions, tests, or configurations) that guide the agent, preventing similar issues in the future. Proactive quality: Instead of reactive code reviews, the focus shifts to designing systems where AI agents are inherently guided towards desired quality standards. This requires robust CI/CD pipelines that can validate agent output without constant human intervention.
Addressing Cognitive Debt in AI-Accelerated Development
Triple Debt Model
Margaret-Anne Storey's Triple Debt Model highlights three types of debt in AI-accelerated development: Technical debt (in the code), Cognitive debt (in people's understanding), and Intent debt (in undocumented rationale and design constraints). Harness engineering primarily tackles cognitive and intent debt by creating a transparent and controlled environment for AI agents.
A significant challenge with AI-generated code is the potential for cognitive surrender, where developers lose track of how the system is built, making maintenance and debugging difficult. Harness engineering mitigates this by providing mechanisms to understand what AI agents are building without needing to inspect every line of code. This includes: Enforcing quality conventions: Automatically validating code against predefined standards. Architectural Decision Records (ADRs): Providing agents with historical context and rationale for design choices, transforming intent debt into explicit guidance. The goal is to build confidence in the AI-generated output through systematic enforcement and documentation.
Implementing Operability and Leading Metrics
Harness engineering emphasizes operability, defined as the combination of software, infrastructure, and processes. To ensure AI agents contribute to a production-ready codebase, it's crucial to integrate comprehensive monitoring and testing, focusing on both lagging and leading metrics:
- Lagging Metrics: Deliverability metrics like DORA metrics, user satisfaction, business operations, and commercial metrics. These indicate the impact after deployment.
- Leading Metrics: Deployability metrics such as failure rates, performance/scalability testing, failure scenario testing, penetration testing, and compliance auditing. These predict potential issues before they become critical.
AI agents can excel at implementing the tedious but essential checks for leading metrics, provided they have clear definitions of "good." The existing continuous delivery pipeline becomes the operability harness, integrating these sensors to continuously validate the output of AI-driven development. This proactive approach allows organizations to go "all-in" on AI development safely by ensuring a robust system that builds reliable software.