我们如何用3个月提升LLM代码审查准确率
How We Boosted Our LLM Code Review Accuracy by 40% in 3 Months
Last quarter, our LLM-based code reviewer was catching only 52% of critical bugs. Today, it’s at 73%. Here’s exactly how we got there—the playbook, the screw-ups, and the stuff I’d do differently.
I’ll be honest—when we first integrated a large language model into our CI/CD pipeline back in January, I expected magic. Actual magic. What I got instead was a flood of false positives and a missed null pointer exception that took down our payment service at 11pm on a Saturday. That was a fun weekend. By fun I mean my CTO texted me “call me” with no context. You know the vibe.
That’s when it clicked: treating an LLM like a plug-and-play senior engineer is a recipe for disaster. It’s not a senior engineer. It’s not even a junior engineer. It’s a tool, and like any tool, it needs serious calibration.
If you’re leading an engineering team and exploring AI-assisted code review, here’s what actually moved the needle for us. Some of this might be obvious. Some of it surprised me. Actually, wait—I should clarify that most of it surprised me.
1. We Stopped Asking for Everything, Everywhere
Our initial prompt was basically a novel: “Review this code for bugs, security vulnerabilities, performance issues, style violations, and adherence to best practices.” I think we even threw in “and be thorough” at the end. Because that’s how prompts work, right? Just add “be thorough.”
The model skimmed the surface on all five areas and excelled at exactly none. Classic garbage-in-garbage-out situation.
So we pivoted to a multi-pass architecture. Three separate passes, each with a hyper-specific job:
- **Pass 1:** Security vulnerabilities only. OWASP Top 10 focus, nothing else. We included 8 real examples from our own codebase where we’d been burned before—SQL injection in the reporting module, XSS in user-generated content, that sort of thing.
- **Pass 2:** Logic bugs and edge cases. Null pointers, off-by-one errors, state machine violations. This one catches the stuff that keeps you up at night.
- **Pass 3:** Performance anti-patterns. N+1 queries, memory leaks, blocking I/O on the main thread. The things that don’t break your app immediately but make it crawl at 3am when traffic spikes.
Each pass uses its own specialized prompt with few-shot examples pulled from our actual PR history. Not synthetic examples. Real ones.
This alone took our bug detection rate from 52% to 64% in about two weeks.
The lesson? Precision beats breadth every single time. Andrew Ng talks about this a lot—narrow AI applications consistently outperform general ones in production. I should probably listen to him more.
2. We Built a Feedback Loop That Actually Closes
Most teams I’ve talked to stop at “the model flagged it, a human reviewed it, done.” That’s what we did for the first month too. And we stayed flat at 64%.
So we built something we call a review registry. It’s honestly just a Postgres table—nothing fancy. But it logs every single false positive and false negative: the code snippet, the model’s verdict, the human reviewer’s correction, and a category tag. We built a little Slack bot that lets reviewers submit corrections in like 3 seconds. If it takes longer than that, people won’t do it. Learned that the hard way.
Every Friday afternoon, our staff engineer runs a fine-tuning script on this growing dataset. We’re not doing full model retraining (we’re not OpenAI, and our GPU budget is... let’s call it “modest”). But we use LoRA adapters on top of our base model. The impact compounds in a way I didn’t expect:
- Month 1: Accuracy improved 5%
- Month 2: Another 4%
- Month 3: Another 3%
By month three, something wild started happening. The model began catching domain-specific bugs—like incorrect state transitions in our order management system—that generic linters would never find. Stuff that requires actually understanding our business logic. That’s when I knew we were building institutional knowledge, not just consuming AI.
Well... that’s complicated. I should say we’re starting to build institutional knowledge. It’s early.
3. We Measured What Actually Matters (And Ignored the Rest)
Early on, I obsessed over precision and recall like everyone else. Spent hours tweaking thresholds. Made very fancy dashboards.
Nobody cared. My CEO certainly didn’t.
So we started tracking three things that actually map to business outcomes:
- **Mean Time to Detect (MTTD):** How quickly a bug is caught after commit. Dropped from 4.2 hours to 1.1 hours. That’s real money saved.
- **Reviewer Fatigue Score:** We survey senior devs every Wednesday on a 1-5 scale. Started at 2.8 (“drained, please make it stop”). Now at 4.1 (“I actually get to think about architecture again”). This one hit home for me personally—I was burning out our best people.
- **Escape Rate:** Bugs reaching production. Down 31% quarter-over-quarter. This is the number that got my CEO to ask “what do you need to scale this?” instead of “why are we spending on AI tools?”
These numbers got budget. The accuracy percentage? That got polite nods in standup.
If you take one thing from this post, make it this: connect engineering metrics to customer value. It’s a lesson from Marty Cagan’s “Empowered” that I keep having to relearn. Probably will relearn it again next quarter too.
4. The Human-in-the-Loop Paradox
Here’s the counterintuitive part. Actually, it’s more than counterintuitive—it’s kind of the opposite of what I expected.
As our LLM got more accurate, we had to increase human oversight.
Why? Because developers started trusting the model too much. Not maliciously—just... automatically. We caught two instances in March where a junior engineer rubber-stamped the AI’s suggestion without understanding the context. One of them would have introduced a race condition in our inventory system. The kind of bug that doesn’t show up in testing but manifests as “why do we have -3 items in stock” at the worst possible moment.
So we now enforce a simple rule: LLM suggestions are “advisory” for junior devs, “confirmatory” for seniors. Every AI-flagged issue must include a human comment explaining the “why” before merge. Not just “fixed” or “done.” An actual explanation.
This slowed us down by maybe 8% initially. Some seniors pushed back. I get it—adding process feels bad. But our escape rate dropped further, and honestly? The junior devs are learning faster because they have to articulate the reasoning.
Speed without safety is just technical debt with a pretty interface. I think I stole that from a conference talk. If it was yours, tell me and I’ll buy you coffee.
What’s Next
I’m not sharing this because we’ve cracked the code. We haven’t. Our accuracy is 73%, and I want it at 85% by end of year. That’s probably ambitious. Actually, it’s definitely ambitious. But the path is clearer now: narrow the scope, close the feedback loop, measure business impact, and never fully automate judgment.
What’s your experience with LLM-based code review look like? Have you found a sweet spot between automation and oversight? I’m genuinely curious—drop your numbers or horror stories in the comments. Especially the horror stories. Those are always more useful than the success stories somehow.
#AIEngineering #CodeReview #LLM #DevOps #EngineeringLeadership
读者评论 3