AI Code Review Metrics: How to Tell If It Is Actually Working

The dashboard that lies to you

Every AI code review tool ships a dashboard, and most of them measure the wrong thing. "12,438 PRs reviewed." "31,207 findings surfaced." These are activity metrics, and activity is the one thing an AI reviewer produces for free. A tool that posts eleven comments on every two-line fix will top every activity chart while your senior engineers quietly mute it.

Whether AI review is workingis a different question: is the code better, is the team faster, and is the spend justified? Those have real metrics — five of them, in our experience — and none is "number of comments." This post walks through each one: what it tells you, how to instrument it, and the failure mode it catches. If you are still rolling out, start with the 30-day rollout playbook — these metrics are what you graduate to once reviews are flowing.

1. Acted-on rate: the signal-quality number

The acted-on rate is the share of AI comments that produce a response: a code change, a substantive reply, or an explicit "dismissed because" — anything except silent scrolling-past. It is the closest thing to a single quality score an AI reviewer has, because it prices every finding at what the team thinks it is worth, not what the model thinks.

• How to measure: sample 50-100 recent AI comments monthly and sort them into acted-on / ignored / wrong. Thirty minutes of tagging beats any automated proxy, though resolve-and-dismiss counts from your review tool get you most of the way.
• What it catches: the correct-but-useless reviewer. A finding can be technically right and still be noise — if nobody will ever act on a class of finding, suppressing it is respecting your own decisions, not lowering standards.
• What to do with it: tune toward your noise budget. Suppress or reroute the categories that are dragging the rate down, and route sharper models at the paths where findings do get acted on.

2. Time-to-merge: the velocity check (expect a dip)

The business case for AI review is usually speed: mechanical issues get caught in minutes instead of waiting a day for a human pass. So track median time from PR opened to merged, segmented by PR size, before and after adoption.

The trap is judging week one. A new AI reviewer initially slows merges, because it surfaces issues that used to sail through unseen — that is the tool doing its job. The pattern you want over 60-90 days is a dip, then recovery, then improvement past the old baseline as reviewers stop doing mechanical passes and PR authors stop waiting on them. A time-to-merge that never recovers usually means findings are landing too late, too verbosely, or on changes that never needed deep review — a model-routing problem more often than a model problem.

3. Escaped defects: the ground truth

The point of review is fewer bugs in production, so close the loop from the incident side. Each time a production bug traces back to a reviewed diff, tag it with one of three labels: the AI flagged it and the finding was dismissed; the AI missed it; or it was outside review scope entirely.

The split is the diagnosis. Flagged-but-dismissed escapes are a trust problem — the signal existed and the process ignored it, which argues for tuning noise down so the important findings stop drowning. Missed escapes are a capability problem — the paths where they cluster are where premium models belong. This is the slowest metric to accumulate and the only one that measures the thing you actually bought the tool for; even a simple label in your incident tracker builds a decisive picture within a quarter.

4. Human reviewer load: the redistribution you were promised

AI review does not replace human review — with the trust numbers where they are (the 2025 DORA report found roughly 30% of developers have little or no trust in AI-generated code), it could not. What it should do is redistribute human attention: fewer rounds spent on null checks and error handling, more on intent and design.

• Review rounds per PR should fall — mechanical issues get fixed before the first human pass, so fewer back-and-forth cycles.
• Time from PR open to first human review should fall — the AI first pass means reviewers pick up PRs that are already clean.
• The distribution of review load across seniors should flatten. If two staff engineers were the review bottleneck before, this is where relief shows up — a concern we dug into in the review-bottleneck post.

A subjective check is worth adding: ask reviewers quarterly whether the AI first pass makes their review easier or noisier. One question, five-point scale. Sentiment leads behavior — it predicts muting before it happens.

5. Cost per acted-on finding: the number for the budget meeting

Total AI review spend is not a metric; it is a bill. The ratio that means something is spend divided by acted-on findings — what you are paying per finding the team considered worth acting on. It catches both failure modes at once: the expensive reviewer nobody listens to (numerator up, denominator down) and the cheap one that finds nothing (both near zero).

Driving it down is mostly routing and budgeting, not model shopping: lean models on docs, tests, and dependency bumps; premium models on auth, payments, and public APIs; hard caps so an anomalous month pauses reviews instead of overspending. On PURA the accounting side is deliberately boring — BYOK means inference runs on your own API keys at provider rates, and per-repo and per-team budgetsput a floor under the denominator's worst month. If you cannot compute this ratio from your current tool's billing, that is itself a finding.

Putting it together: a one-page scorecard

Five metrics is a scorecard, not a dashboard project. A monthly one-pager — acted-on rate, time-to-merge trend, escaped-defect tags, reviewer-load deltas, cost per acted-on finding — fits in a team meeting and answers the three questions that matter: is the signal good, is the team faster, is the spend sane.

Two closing rules. First, decide the targets before you look at the data — a noise budget and an acceptable cost ceiling agreed in advance turn tuning debates into arithmetic. Second, never wire these numbers to individuals. The moment acted-on rate becomes a per-developer KPI, developers start acting on findings to move the number, and the metric stops measuring the tool. Measure the system, tune the system, and let the reviewer earn its place on the team the same way anyone else does: by being worth listening to.