AI Code Review for Feature Flags: Inverted Defaults, Forgotten Readers and the Cleanup That Deletes the Wrong Branch

Feature flags are the mechanism teams use to make deploys safe, and they are also the mechanism behind a specific family of outages: the flag that was on in staging and off in production, the default that was inverted in a refactor, the kill switch that killed the wrong thing, and the flag that was removed from the config a year after the code that read it stopped existing. None of those bugs look like bugs in the pull request. They look like a one-line change to a boolean.

This post is a checklist for what an AI code reviewer should check on a pull request that adds, flips or removes a feature flag, and how to turn the checklist into review instructions for whichever tool you run. Where PURA does something specific we say so; most of it applies to any reviewer, human or model. It follows the same shape as our checklists for database migrations, infrastructure as code and dependency updates, all of them changes whose risk is mostly outside the diff.

Why flag changes defeat the default review

A flag PR is usually tiny and usually boring. It adds an if (flags.newCheckout) around a code path, or changes a default from false to true, or deletes a branch that has been dead since the flag went to one hundred percent. A reviewer that reads only the diff sees nothing wrong, because nothing is wrong in the diff. The risk lives in three places the diff does not show: the configuration that sets the flag in each environment, the other code that reads the same flag, and the sequencing between this deploy and the next one.

That is the general failure mode we described in why diff-only reviewers miss real bugs, and feature flags are one of its cleanest examples, because the whole point of a flag is to decouple the code change from the behaviour change. A review that only looks at the code change is reviewing half of it.

The checklist

1. What is the default, and is it the same in every environment?

The first question on any flag PR is what happens when the flag is missing. Code that does flags.get('newCheckout') and treats undefined as falsy has one behaviour; code that does flags.get('newCheckout', true) has the opposite. A reviewer should state the resolved default in plain words and check it against every environment config in the repository: the local dev defaults, the test fixtures, the staging and production values if they are checked in, and the fallback in the flag client's initialisation. An inverted default is the most common flag bug and the hardest to see, because the diff shows a boolean and the bug is in a file that did not change.

2. Is the flag read in exactly one place?

Flags spread. The first PR reads it in the request handler; the third reads it in a background job; the fifth reads it in a template. Once a flag is read in several places, flipping it no longer flips one behaviour, it flips several, and some of them were written by people who assumed the flag meant something slightly different. The reviewer should search the repository for every read of the flag name and list them, and should flag a PR that adds a second reader without a comment explaining why the first one cannot be reused. This is a repository-wide check; a reviewer that sees only the diff cannot do it.

3. Does both sides of the branch still work?

A flag guards two code paths, and the one that is off in your environment is the one that rots. On a PR that touches a flagged area, the reviewer should check that the tests exercise both the on and the off path, not just the one the author ran. The tell is a test file that sets the flag once at the top and never varies it. The same applies to the kill switch case: if the point of the flag is to turn the new thing off in an emergency, somebody should have proven that the old thing still functions when it is.

4. Is the flip sequenced correctly with the deploy?

Flags that gate a schema change, a new message format or a new API field have an ordering problem that is exactly the one we wrote about for migrations. If this PR both adds the code and flips the flag on, then during a rolling deploy some instances will serve the new behaviour and some the old, and anything stateful between them, a queue, a cache, a shared table, sees both at once. The reviewer should ask whether the flag is being flipped in the same PR as the code it guards, and if so, whether the two behaviours are compatible with each other while both are running. Usually the answer is to land the code dark and flip in a separate change.

5. Does removing the flag remove the whole flag?

Cleanup PRs are where the subtle bugs live. Removing a flag means deleting the conditional, deleting the dead branch, deleting the config entries in every environment, deleting the test variants, and deleting the flag definition in the flag service. A PR that does three of those five leaves either dead code or, worse, a config key that still exists and that a future reader will assume still does something. The reviewer should enumerate the five and check each one off, and should look especially hard at which branch was kept: the number of cleanup PRs that keep the old path and delete the new one is not zero.

6. Is the flag name and lifecycle recorded?

A flag with no owner and no planned removal date becomes permanent configuration. The reviewer should check that a new flag has, somewhere the team can find it, a one-line purpose, an owner and an intended end state (permanent operational switch, or temporary rollout flag to be removed by a date). Teams that do this in a flags registry file in the repository give the reviewer something concrete to verify; teams that do it in a wiki give it nothing, and the review instruction should say to ask for it in the PR description.

7. Are the mechanical details right?

• The flag name is spelled identically in the code, the config and the flag service, with the same case. A typo produces a flag that is silently always off.
• The flag is evaluated once per request or job, not once per process at startup, unless startup evaluation is intended. A flag cached at boot cannot be used as a kill switch.
• User-targeted or percentage flags use a stable key, so the same user gets the same behaviour across requests; a flag keyed on a random value is an A/B test nobody designed.
• Logging and metrics include the flag state, so that an incident can be sliced by which path the request took.
• Secrets and environment-specific values are not being smuggled in as flag values.

Turning this into review instructions

Match the PR, not the whole repo. Flag PRs are easy to detect: a change to the flags registry, to any file under the flag configuration directory, or a diff that adds or removes a call to the flag client. A path-based or pattern-based rule in the review agent's skill can attach the checklist above to those PRs and leave the rest alone, so the ordinary PR does not get seven flag questions it does not need. Custom rules that actually work covers how to write that rule so it stays specific.

Give the reviewer the configuration. The checks that matter most, default resolution and the list of readers, need the whole repository. In PURA the review agent has the repository available, so an instruction such as “for every flag read in this diff, list all other reads of the same flag and the resolved default in each checked-in environment config” produces a concrete answer rather than a reminder to be careful. If your flag values live outside the repo, in a flag service's dashboard, say so in the instruction so the reviewer asks for them instead of guessing.

Decide what blocks. Most flag findings are advisory. Two are worth gating on: an inverted or inconsistent default across environments, and a cleanup PR that removes the flag from code but not from config. Both are cheap to check and expensive to ship. Gating versus advisory review has the general framework.

Route cleanup PRs to a stronger model. Flag removals are rare, touch many files, and have the highest ratio of risk to diff size. They are a sensible place to spend a better model, in the same way migrations are; our post on budgets by provider and model describes how to do that per repository without paying for it on every PR.

What the reviewer still cannot do

A code reviewer cannot see the value of a flag in a service that is not in the repository, cannot know that the rollout is at thirty percent rather than one hundred, and cannot tell you whether the person who asked for the flag still works there. It can tell you what the code will do under each possible value, which readers exist, and whether the cleanup was complete. Treat it as the part of the review that checks the mechanics exhaustively, so the human reviewer can spend their attention on whether the flag should exist at all.