Meta's JiTTesting: Scalable AI-Powered Test Generation with Human-on-the-Loop
Meta's JiTTesting system leverages large language models (LLMs) to automatically generate Just-in-Time Tests (JiTTests) for code changes, aiming to catch bugs before they land in production. This architecture integrates multiple workflows and assessment stages, including LLM-as-Judge and a human-on-the-loop approach, to ensure high true-positive rates and maintain developer velocity at an extreme scale.
Read original on Meta EngineeringMeta Engineering's JiTTesting system is a sophisticated approach to automated bug detection, crucial for managing the immense scale and rapid evolution of their codebase. Unlike traditional AI-based code reviews, JiTTests generate executable tests that specifically expose behavioral differences introduced by a code change. This system is designed to proactively identify risks and significantly reduce the occurrence of severe bugs, particularly privacy-related ones, before they impact users.
JiTTesting Architecture and Workflows
The core of Catching JiTTests involves two tandem workflows for generating tests targeting code changes (diffs):
- Intent-Aware Workflow: Infers the intention and potential risks of a code change, then targets test generation towards likely failure points by constructing mutants that represent possible bugs.
- Dodgy Diff Workflow: Treats every code change as if it were a mutant and generates tests to expose behavioral differences without reasoning about intent, aiming to maximize the pool of potential bug catches.
Automated Assessment and Human Integration
To filter false positives and ensure actionable insights, the system employs a multi-stage assessment process:
- LLM-as-Judge Ensemble: Aggregates judgments from multiple state-of-the-art LLMs to classify weak catches as likely true or false positives, reviewing test code, execution outcomes, diffs, and inferred intent.
- Rule-Based Assessor (RubFake): A deterministic, rule-based approach complements LLM-as-Judge, providing verified judgments and helping rank weak catch candidates.
- Human-on-the-Loop: Instead of engineers reviewing every output, this approach focuses on aggregate-level review and criteria setting, intervening at the policy level to reduce friction while maintaining human oversight and label consistency.
System Design Takeaway
The JiTTesting system demonstrates a successful application of AI at scale, not just in test generation but in intelligently managing the testing workflow. Its ability to combine non-deterministic LLM judgments with deterministic rule-based checks and human-on-the-loop validation is key to its high true-positive rate and engineer engagement.