AI Code Review for Configuration and Environment-Variable Changes: The Checklist Before a Value Ships

One line, no tests, and the outage that only happens in production

A configuration change is the pull request everyone approves on the way to lunch. It adds PAYMENT_TIMEOUT_MS to a settings module, bumps a value in .env.example, renames REDIS_HOST to CACHE_HOST because the cache is no longer Redis, or sets a flag in a Helm values file. The diff is a handful of lines, the tests pass because no test reads the environment, and the value in the diff is not even the value that will run in production. That last part is the whole problem. Code is reviewed on what it does; config is reviewed on what the author says it will do, and the gap between those shows up as the service that starts in staging and dies in production, the feature that is on where it should be off, or the secret that is now in git history forever.

This post is the checklist for that category: eight questions, the failure each one prevents, what an AI reviewer can verify from the diff and the repository, what it can only ask, and the rules that make the questions run on every change to an env file, a settings module, a compose file, a CI workflow or a values file. It sits next to our checklists for infrastructure as code, feature flags and logging and observability changes, and it follows the same rule as all of them: the risk is rarely inside the changed lines.

Question 1: Is a secret being committed, echoed or drifting?

The obvious version is a real credential in a committed .env, a docker-compose.yml with a database password in plain text, or a CI workflow that sets a token as a literal instead of referencing a secret. The less obvious versions are the ones that pass a quick glance: a .env.example that was copied from a working .env and still carries a live key, a Helm values file with a default password that was meant to be overridden, and a startup script that prints its environment for debugging and so writes every secret to the build log.

A reviewer with repository context can do three things a diff-only reviewer cannot: match values against the shapes of known credential formats, check whether the file is covered by .gitignore and whether a committed copy already exists, and compare .env.example against the keys the code actually reads so the example neither leaks a value nor falls behind. What it cannot know is whether a string that looks like a key is a placeholder; the right comment names the line and asks, and if it is real, the answer is rotation, because removing it from the diff does not remove it from history.

Question 2: Does every environment get the new key?

The classic config outage is a new required variable that was set on the developer's machine, set in staging during testing, and never set in production. The code reads it, gets undefined, and either crashes at startup, which is the good outcome, or carries on with an empty string until the first request that needs it. The diff shows the new read; it does not show the deployment manifests, the secret store or the platform dashboard where the value has to be added.

Where those manifests live in the repository, as they do with compose files, Kubernetes manifests, Helm values per environment and most CI configuration, the reviewer can list every environment definition and report which ones have the key and which do not. Where the value lives in a hosted secret manager or a platform console, it can only say that a new required key exists and ask whether it has been set everywhere, which is still the comment that stops the deploy.

Question 3: Does the string become the type and unit the code expects?

Environment variables are strings, and every language has its own way of pretending otherwise. The string 'false' is truthy in JavaScript and Python; FEATURE_ENABLED=false read with if (process.env.FEATURE_ENABLED) turns the feature on. The string '0' is also truthy. An unset variable and an empty one are different things to some loaders and the same thing to others. A timeout named _TIMEOUT with no unit is read as seconds by the author and set in milliseconds by the operator, and the request waits for thirty minutes.

All of this is visible in the diff and the surrounding code. The reviewer can follow each new key to the line that reads it, check whether the value is parsed with an explicit boolean or integer conversion, and compare the unit implied by the name with the unit used at the call site.

Question 4: Is the value validated at startup or discovered at first use?

A service that validates its configuration once, when it starts, fails loudly in the deploy pipeline where someone is watching. A service that reads each variable lazily, at the point of use, fails at 3 a.m. on the first request to the one code path that needed it. Settings schemas, whether a zod object, a pydantic settings class, a Go struct with a validation pass or a hand-written function, exist to move that failure to the front, and a PR that adds a key without adding it to the schema has quietly opted out.

The reviewer can see whether the repository has a central settings module and whether the new key went through it or around it. A bare process.env read deep in a handler gets the comment to route it through the schema with a type, a default or a required marker.

Question 5: Does the old name still have readers?

Renames and removals are where config drifts into the failure mode of feature flags: the name changes in the settings module, the new name is set everywhere, and a script, a cron job, a second service in the monorepo or a Dockerfile is still reading the old one. Nothing errors; the old reader gets a default and behaves differently from the rest of the system. Removal is the same story with a sharper edge, because the leftover reader now gets nothing.

This is the question a reviewer with the whole repository answers best. It can search for every occurrence of the old name across code, shell scripts, compose files, workflows and documentation, and list what the PR did not touch.

Question 6: What wins when the same key is set twice?

Most loaders have a precedence order, and most authors have a mental model of it that is slightly wrong. Does a .env file override a variable already present in the shell, or the other way round? Does .env.local beat .env? Is the value in the CI workflow's env: block overridden by a repository secret with the same name, or does it override the secret? A PR that adds a default to a file while the same key is set in the deployment environment may change nothing, or may silently replace the production value, depending on the loader.

The reviewer can identify the loader from the repository, state its documented precedence, and point out every place the new key is now set. It cannot know which of those the author intended to win, so the comment should present the order and the places and ask which one is the source of truth.

Question 7: Does this config deploy with the code, or somewhere else?

A code change ships when the pipeline runs. A config change may ship with it, or it may be a value in a platform console that someone has to change by hand, before or after the deploy, and the ordering matters: a reader that requires the new key must not go live before the key exists, and a key removed from the store must not disappear before the last reader is gone. The PR description rarely says which it is. Teams that keep configuration as code in the repository get the ordering for free; teams that do not need the ordering written down on the PR.

The reviewer can tell from the repository whether the value is defined in a tracked file or is only read, and in the second case it should ask where the value lives, who sets it in production, and whether the change has to happen before or after this PR merges.

Question 8: Is the value dangerous even though it is not a secret?

A configuration review that only looks for secrets misses the values that cause the most public incidents. DEBUG=true in a production profile exposes stack traces and sometimes an interactive debugger. A CORS allowed-origins value of *, or a regex that matches more than it should, turns a private API into a public one. A database URL with the credentials embedded in it is a secret wearing a non-secret name. A connection timeout of zero means forever in some clients and immediate failure in others. A retry count raised to make an integration test pass becomes a retry storm against a real upstream.

Each of these is readable in the diff, and the reviewer can flag the pattern: a debug or verbose flag enabled in a non-development profile, a wildcard or very broad origin, a credential inside a URL, a zero or very large timeout, a retry or concurrency limit that moved by an order of magnitude. Whether the value is a deliberate trade-off is the team's call, so the comment quotes the old and new values and the consequence and asks.

Turning the checklist into rules

Eight questions are too many to remember on a one-line PR, which is the case for writing them down as review rules that run whenever a change touches an env file, an example env file, a settings module, a compose file, a workflow, a Dockerfile ENV line or a values file. PURA's custom rules are written in plain language, and for this category they look like this.

• Block any committed value that matches a credential format in an env, compose, workflow or values file, and flag any .env.example key whose value is not an obvious placeholder; flag startup scripts that print the environment.
• For every new key the code reads, list the environment definitions in the repository that set it and those that do not, and require either a default or a startup validation failure.
• Flag any environment variable compared truthily without an explicit boolean parse, any integer read without conversion, and any timeout or interval whose name does not state a unit.
• Flag reads of the environment outside the central settings module, and new keys that are not added to the schema.
• On a renamed or removed key, search the whole repository for the old name and list every remaining reader, including scripts, workflows and documentation.
• When a key is set in more than one place, state the loader's precedence order and ask which location is the source of truth.
• When a key is read but defined nowhere in the repository, ask where it lives in production and whether it must be set before or after the merge.
• Flag debug flags in non-development profiles, wildcard or broad CORS origins, credentials embedded in URLs, zero or extreme timeouts, and retry or concurrency limits that change by an order of magnitude, quoting old and new values.

PURA runs on your own infrastructure and reviews with your own OpenAI, Anthropic or Google keys, which is the right shape for this category: the rules describe how your configuration is laid out and the reviewer reads your env files and manifests, and none of that leaves your boundary. The flat platform fee is metered by reviews, from Starter at $39 for 100 reviews through Scale at $99 for 400 to Pro at $199 for 1,000, with Enterprise above that and budgets per team, repository and provider, so a rule set that runs on every config PR costs the same whether it fires or stays quiet. The reviewer does not decide what the production timeout should be; it makes sure the question is on the PR before the value is live.

The one-paragraph version

A configuration change has no tests and ships a value that is not the value in the diff, so review it as a deployment, not as code. No secret committed, echoed or left in an example file; every new key set in every environment or guarded by a default or a startup failure; strings parsed into the intended type and unit; the read routed through the settings schema; no remaining readers of an old name; a stated precedence when a key is set twice; a stated home and ordering when the value lives outside the repository; and a second look at the values that are dangerous without being secret. Five of those a reviewer with repository context can verify outright; three it must ask. Write all eight as rules, and the one-line PR gets the review its blast radius deserves.