PilotPlanPHASE 01
Source-backedEvaluation set
- Collect real and synthetic pull requests with verified bugs
- Define severity, relevance, and acceptable-noise measures
Exit gate: Evaluation set reviewed by experienced engineers
Documented implementation, not a PilotPlan customer story
GraphiteAI code review
How Graphite implemented AI-powered code review
A documented AI code review case covering model evaluation, multi-step analysis, validation layers, developer feedback, and reported adoption outcomes.
The problem
Engineering teams needed faster pull-request feedback without flooding developers with subjective or low-value AI comments.
How it was implemented
- 01Evaluated models against 500 synthetic and real pull requests with known bugs.
- 02Split code analysis into discrete steps instead of relying on one undifferentiated prompt.
- 03Added multiple validation layers, including voting and self-critique, before comments reached developers.
- 04Focused comments on objective bugs and generated fix suggestions for developer review.
Reported outcomes
Reported by Anthropic and the featured company. Not independently verified by PilotPlan.
- Graphite reported a 40-times faster feedback loop, from about one hour to 90 seconds.
- The company reported a 96 percent positive-feedback rate on AI comments.
- Sixty-seven percent of AI suggestions reportedly led to code changes.
What another team can learn
- Build evaluations from the actual artifact and errors the system must detect.
- Decompose complex review into stages and filter low-confidence output.
- Developer acceptance and resulting code changes are useful signals, but not substitutes for escaped-defect measurement.
- Keep humans responsible for merge decisions and security-sensitive changes.
Technical architecture
A simplified logical architecture reconstructed from the public case study. It is not claimed to be the company's private network diagram.
The 500-pull-request evaluation, discrete analysis steps, voting, self-critique, objective-bug focus, and fix suggestions are documented. Confidence thresholds and feedback storage are a reasonable reconstruction of the described validation system.
Rollout plan
A practical sequence based on documented milestones where available, with inferred and recommended steps clearly marked.
PHASE 02
Source-backedAnalysis pipeline
- Break review into discrete tasks
- Use voting and self-critique to filter comments
Exit gate: Precision threshold met on unseen pull requests
PHASE 03
InferredDeveloper pilot
- Release to selected repositories
- Capture positive feedback, dismissals, and resulting changes
Exit gate: High-signal feedback without workflow disruption
PHASE 04
RecommendedProduction scale
- Expand repository and language coverage
- Monitor false positives, missed defects, latency, and rate limits
Exit gate: Operational SLO and incident ownership established
Components and integrations
What the implementation needs, and how confidently the public evidence supports each element.
Synthetic and real pull-request evaluation set with known bugs
Multi-step Claude analysis for code understanding
Voting and self-critique validation layers
Developer comments, one-click suggestions, and feedback loop
Team and responsibilities
The accountable roles needed to build, approve, and operate this kind of system.
AI engineering owns evaluation, orchestration, and comment quality
Developers review findings and decide whether to change code
Security and language specialists review high-severity evaluation sets
Security and operating controls
Controls explicitly documented or required to make the reconstructed implementation safe enough to operate.
Restrict automated comments to high-confidence, actionable findings
Keep merge authority with humans and existing branch protections
Prevent repository code, secrets, and prompts from leaking across tenants
Reliability and failure handling
What should happen when the model, integration, downstream system, or generated output is wrong.
Noisy comments: raise thresholds or suppress the affected rule
Missed severe bug: add it to the evaluation set and rerun regression tests
Model or rate-limit failure: allow normal human review to continue without blocking
Success metrics
Published measures are separated from the additional metrics a responsible implementation should track.
Feedback-loop time, positive-feedback rate, and suggestion implementation
Precision, recall, severity-weighted misses, and escaped defects
Developer review time, comment dismissal, latency, and cost per useful finding
Assumptions and unknowns
Public case studies rarely disclose full architecture, permissions, evaluation data, cost, or failure rates. These gaps must be validated before treating this as an implementation specification.
- The source explains validation concepts but not every internal service or threshold.
- Positive feedback and accepted suggestions do not equal lower production defect rates.
- The diagram represents the logical pipeline, not Graphite's verified infrastructure topology.
What the source does not prove
- The metrics are reported by Anthropic and Graphite in a vendor customer story.
- Positive feedback and accepted suggestions do not alone prove fewer production defects.
- The source does not provide full false-positive, false-negative, security, or cost data.
Primary source
PilotPlan summarized the implementation and added practical analysis. Read the original vendor-produced case study before relying on any claim.
Graphite AI code review case study by AnthropicQuestions about the Graphite implementation
How did Graphite evaluate AI code review?
The source describes a 500-pull-request evaluation set containing synthetic and real examples with known bugs, followed by user feedback and change-acceptance measurement.
How did Graphite reduce noisy comments?
It decomposed analysis into steps, used multiple validation layers, and focused on objective bugs instead of subjective style suggestions.
What should a code review pilot measure?
Measure false positives, missed defects, severity, developer acceptance, review time, escaped defects, security findings, cost, and the time humans spend validating output.
Build an implementation plan for your actual workflow
Describe the challenge, constraints, current stack, budget, and timeline. PilotPlan researches the options and assembles a sourced implementation plan.