A resource for design system and engineering teams
AI codereviewchecklist.
For pull requests where a model wrote the first draft.
Scripts first. Then people. Nobody grades their own work.
Generated code fails in its own ways.
A person who guesses an API usually knows they guessed. A model does not flag its guesses: the invented prop, the package that was never published and the test that checks nothing all arrive with the same confidence as the correct lines. In a controlled study, developers with an AI assistant wrote less secure code and were more likely to believe it was secure [S11].
Some of these failures are cheap to catch mechanically, and one of them is a supply-chain risk. Across 576,000 generated code samples, 19.7% of the packages the models suggested did not exist, and 43% of those hallucinated names came back in all ten repeated runs of the same prompt [S04]. A name that repeats is a name somebody can register.
This guide orders the review so that scripts go first and human attention goes where only humans help. The kit has a pull request template with the same item numbers and a checker that finds hallucinated imports, invented props and empty tests. On the fixture pull request it reports 11 errors; on the clean version of the same feature, none.
Practical guidance, not a standard. The checker is a scanner, not a compiler: it complements your typecheck, lint, tests and security tooling and replaces none of them. WCAG references are to WCAG 2.2. Prepared with AI assistance and edited by hand.
Start here
Pick your route, then read the labels.
Run the checker on both fixtures first. The two versions of the same dialog show every class of problem this guide covers.
Reviewing one pull request
Run check-ai-diff.mjs on the changed files, then work Sections 01 to 07 in order. Stop at the first section that fails and send it back.
Setting up the gate
Add the PR template, run the checker next to typecheck, lint and tests, and make its errors blocking. Warnings go to the reviewer.
Agents opening pull requests
The same checks apply, with two additions: C06 (a reviewer who did not write it) and C32 (the PR must not widen its own permissions).
Deciding what to automate
Field Guide 21 scores the import and prop checks as automate and the review itself as assist. This guide is where that line sits.
Read the labels before the checks.
| Label | Meaning |
|---|---|
SCRIPT | Found by check-ai-diff.mjs or another tool named in the evidence line. |
REVIEW | Needs a person reading the code or using the feature. |
SECURITY | Traces to a risk in the OWASP Top 10 for LLM Applications 2025. |
WCAG A / AA | Traces to a WCAG 2.2 success criterion at that level. |
PRACTICE | A working method with a review signal rather than a hard gate. |
Section 01
Check the pull request before the code.
Six checks decide whether this pull request is ready for anyone's time.
Suggested owners: Author + reviewer
The PR says what a tool wrote
Tool, model, task and the generated parts. Reviewers read generated code for different mistakes.
Evidence: The AI section of the PR template is filled in.
Recommended practice.
The author ran it and can explain it
The author used the feature and can explain any line. "The model wrote that" is not an answer.
Evidence: Template checkboxes; answers in review without regenerating.
Claude Code docs: if you cannot verify it, do not ship it. S09.
The diff does what the ticket asks, and no more
Agents over-deliver: extra files, abstractions, refactors. Out-of-scope changes get their own PR.
Evidence: Every changed file maps to the ticket.
GitHub Docs: verify context and intent. S07.
One concern per pull request
Generated refactors apart from behaviour changes, dependency changes apart from both.
Evidence: The PR title names one change.
Recommended practice.
Machine checks run before a person reads
Typecheck, lint with token rules, tests and
check-ai-diff.mjspass, with output in the PR.Evidence: The Machine checks block in the template.
GitHub Docs: start with functional checks. S07.
The reviewer did not write it
Another person, or a reviewer agent in a fresh context. Never the session that wrote the code.
Evidence: Reviewer field in the template.
Claude Code docs: review in a fresh context. S09.
Review time is the scarcest input in the pipeline. Spend it on ready code.
Section 02
Verify every package.
A hallucinated package name fails the build if nobody owns it, and runs someone else's code if somebody does. Check the manifest first, the registry second.
Suggested owners: Reviewer + platform owner
Every bare import is declared in package.json
An import that is not a dependency is either hallucinated or works only by accident of hoisting. Both fail somewhere else later.
Evidence:
import/undeclared-package: 2 errors on the fixture (@acme/ui-hooks,clsx).OWASP LLM09: models suggest non-existent libraries. S03.
Every new dependency is real and intended
Run
npm view <name>: first publish date, maintainers, repository. A week-old package with a model-suggested name is a warning sign.--registrydoes the lookup.Evidence:
npm viewoutput in the PR;registry/not-foundon the fixture for@acme/ui-hooks.19.7% hallucinated packages, 43% repeated in all ten runs. S04. Slopsquatting. S05.
Nobody installs a package to find out whether it exists
Install runs the package's preinstall and postinstall scripts;
npm viewinstalls nothing. To evaluate, use--ignore-scriptsin a throwaway environment.Evidence: Template checkbox.
npm scripts and config docs. S12, S13. A registered hallucinated name drew over 30,000 real downloads. S06.
Lockfile changes match package.json changes
No new transitive packages without a new direct one, no registry URL changes, no integrity hashes rewritten for unchanged versions.
Evidence: Lockfile diff reviewed; a lockfile lint in CI if you have one.
OWASP LLM03 supply chain. S01.
Check the name against the manifest, then the registry. Never against the installer.
Section 03
Hold imports and props to the contract.
The contract lists what exists (Field Guide 02). Generated code is where things get invented.
Suggested owners: Reviewer + design-system lead
Every relative import resolves
Generated code imports helpers it assumed exist, such as
./lib/analytics.Evidence:
import/missing-file: 1 error on the fixture.GitHub Docs: hallucinated APIs. S07.
Design-system imports use the package entry point
@acme/ui/src/components/Dialogcouples product code to internals that change without notice.Evidence:
import/deep-import: 1 error on the fixture.Field Guide 02 01.02: the exact import is written down.
Every named import is exported
PrimaryButtonsounds right and does not exist. Named imports are checked against the exports.Evidence:
import/unknown-export: 1 error on the fixture.Field Guide 04 W16: name the hallucinations.
Every prop is in the component contract
variant,shadow,className: if the contract does not list it, it does not exist.Evidence:
props/unknown-prop: 2 errors on the fixture.Field Guide 02 02.02: invented props fail validation.
Every literal value is in the prop's enum
size="small"andtone="ghost-primary"are plausible and wrong.Evidence:
props/invalid-value: 2 errors on the fixture.Field Guide 02 01.07: closed sets are enums.
Forbidden combinations fail the typecheck
The Field Guide 02 union types reject forbidden pairs. No
ascasts or@ts-ignorearound them.Evidence:
tsc --noEmitpasses; grep the diff for new casts and ignores.Field Guide 02 02.03.
If the contract does not list it, the code cannot use it.
Section 04
Keep the code on the system.
Code can pass every contract check and still route around the design system. These five checks catch the detours.
Suggested owners: Reviewer + design-system lead
No spread props onto design-system components
<Button {...rest}>forwards whatever the caller passed, and no reviewer or script can see it.Evidence:
props/spread: 1 warning on the fixture.Recommended practice.
No re-created components
A local
PrimaryButton, a styleddivwith a click handler, a hand-rolled modal. Look for new components that overlap the system's exports.Evidence: Reviewer note; a lint rule for raw
<button>in product code if you want one.Field Guide 02 06.07: verify, never invent.
No deprecated APIs
Models learned from older versions. Deprecated props and components come back unless a lint rule rejects them.
Evidence: A
no-deprecated-apisstyle lint rule.Field Guide 02 07.03.
No raw values or core tokens
Colours, spacing and radii come from semantic tokens. Inline
style={{ color: "#d4351c" }}is the common generated shortcut.Evidence: The Field Guide 03 ESLint and stylelint gates pass.
Field Guide 03.
Patterns match the codebase, not the internet
State, data fetching, file layout and naming follow this repository, not what was common in training data.
Evidence: Reviewer note.
GitHub Docs: assess code quality and consistency. S07.
A detour around the system is a fork of it.
Section 05
Check the experience, not only the rules.
No script in this kit sees the fixture's unnamed icon button. axe and a keyboard do.
Suggested owners: Reviewer + accessibility lead
Every interactive element has an accessible name
Icon-only controls above all: with Button,
iconOnlyplusaria-label, enforced by the types.Evidence: axe
button-name; the typecheck foriconOnly.WCAG 2.2 SC 4.1.2 Name, Role, Value. S16.
Native elements before ARIA
No
divwithonClick, norole="button"on a button, no ARIA repeating native semantics.Evidence: Review of new JSX; axe role and nesting rules.
APG: no ARIA is better than bad ARIA. S17.
The flow works with a keyboard alone
Every action reachable, focus visible and returned after a dialog, no
outline: none.Evidence: Template checkbox; an interaction test per keyboard path.
WCAG 2.2 SC 2.1.1, 2.4.3 and 2.4.7. S16.
Inputs have labels and errors are linked
Generated forms lean on placeholders. Every input needs a visible label and linked error text.
Evidence: axe
label; a test foraria-describedby.WCAG 2.2 SC 3.3.1 and 3.3.2. S16.
Results are announced
Saving, deleting and failing produce announced text. The fixture skipped exactly this test.
Evidence: A status message test that is not skipped.
WCAG 2.2 SC 4.1.3 Status Messages. S16.
axe passes and a person checked
Tools find part of the issues: all of them together found 71 percent of 143 planted barriers in a GOV.UK test [S19]. That is a floor.
Evidence: axe failing on violations in changed stories; keyboard pass recorded.
S19.
A passing axe run means no rule fired. It does not mean the dialog works.
Section 06
Treat generated code as untrusted input.
OWASP's 2025 LLM list applies to the code a model wrote and to features that call a model.
Suggested owners: Reviewer + security owner
No raw HTML without a sanitizer
dangerouslySetInnerHTMLrenders whatever the data holds. Sanitize it, or render text.Evidence:
security/raw-html: 1 warning on the fixture.OWASP LLM05. S02. Veracode 2026: XSS handled securely in 15% of relevant tasks. S15.
No dynamic code execution
eval,new Functionand string timers turn data into code.Evidence:
security/dynamic-code.S02.
No secrets, internal URLs or personal data
In code, fixtures, snapshots, prompts and comments. Generated fixtures copy real-looking data.
Evidence: Secret scanning in CI; reviewer note.
OWASP LLM02 sensitive information disclosure. S01.
Model output in the feature is untrusted
If the code calls a model, its output is user input: encode, validate, least privilege.
Evidence: Threat notes in the PR for any model-calling feature.
OWASP LLM01 prompt injection and LLM06 excessive agency. S01.
The PR does not widen its own permissions
CI, lint, compiler, coverage or CODEOWNERS changes need a reason. Agents sometimes edit the check.
Evidence: Template checkbox; CODEOWNERS on those paths.
GitHub Docs: watch for tests deleted or skipped instead of fixed. S07.
Automated security review runs on trusted PRs only
A diff can carry instructions. Anthropic's security review action is not hardened against that.
Evidence: Workflow requires approval for external contributors.
S10.
The code a model writes is input to your system. Review it like input.
Section 07
Make the tests prove something.
The essay behind this guide describes 200 generated tests that all passed and tested nothing useful [S18]. Green is a claim; a failing test is evidence.
Suggested owners: Reviewer + engineering lead
Every test asserts behaviour
A test without
expectorassertpasses by running. A test that compares a literal with a literal passes by definition.Evidence:
test/no-assertionandtest/tautology: 1 error each on the fixture.Kit script.
Assertions check values, not existence
toBeDefined(),toBeTruthy()and a lone snapshot prove that something rendered. Assert the role, the name, the call and its arguments.Evidence:
test/existence-only: 1 warning on the fixture.S18: tests that checked functions returned values, not correct values.
No test was skipped, deleted or loosened
.skip, a removed case, a widened snapshot or a lowered threshold to get to green.Evidence:
test/skipped: 1 warning on the fixture; the test diff read line by line.GitHub Docs: tests deleted or skipped instead of fixed. S07.
Each test fails when its behaviour breaks
Break the code once and watch the test fail, or run mutation testing: a surviving mutant marks a test that asserts too little.
Evidence: Mutation score for the changed files, or the author's note of the deliberate break.
Stryker: surviving mutants and mutation score. S14.
Empty, loading, error and long content are covered
Generated tests favour the happy path. Name the edge cases in the PR, tested or listed as follow-ups.
Evidence: Test names or a follow-up list.
The essay's review list: null, undefined, empty, loading. S18.
A test you have never seen fail has not told you anything yet.
Appendix A
Two versions of one dialog.
fixtures/clean and fixtures/ai-pr implement the same delete-project dialog. Output below is from the kit, run on 24 September 2026.
terminal
$ node scripts/check-ai-diff.mjs --root fixtures/clean
3 file(s), 0 error(s), 0 warning(s)
$ node scripts/check-ai-diff.mjs --root fixtures/ai-pr --registry
warn ...test.tsx:6 test/existence-only "renders" only checks that something exists
error ...test.tsx:11 test/no-assertion "calls onDelete" asserts nothing
error ...test.tsx:17 test/tautology "works" compares a literal with a literal
warn ...test.tsx:21 test/skipped "announces the result" is skipped
error ...Dialog.tsx:6 import/unknown-export @acme/ui does not export PrimaryButton
error ...Dialog.tsx:7 import/deep-import @acme/ui/src/components/Dialog: past the entry point
error ...Dialog.tsx:8 import/undeclared-package @acme/ui-hooks is not in package.json
error ...Dialog.tsx:9 import/undeclared-package clsx is not in package.json
error ...Dialog.tsx:12 import/missing-file ./lib/analytics does not resolve to a file
error ...Dialog.tsx:23 props/unknown-prop <Button> has no prop variant in its contract
error ...Dialog.tsx:23 props/invalid-value size="small" is not one of sm, md, lg
error ...Dialog.tsx:29 props/invalid-value tone="ghost-primary" is not one of primary, ...
warn ...Dialog.tsx:32 props/spread <Button {...rest}> cannot be checked against the contract
error ...Dialog.tsx:32 props/unknown-prop <Button> has no prop shadow in its contract
warn ...Dialog.tsx:28 security/raw-html dangerouslySetInnerHTML: confirm it is sanitized
warn (registry) registry/not-found @acme/ui-hooks is not on the npm registry
3 file(s), 11 error(s), 5 warning(s)Shortened for print. Without --registry the run makes no network requests and reports 4 warnings. What it does not see in the same file: the trigger Button has no accessible name (C22), Delete does not say what is destroyed and the dialog has no title. That is the reviewer's half.
Do
- Run it on changed files:
git diff --name-only origin/main... - Keep contracts next to components and point
--contractsat them - Generate
design-system.jsonfrom your package's real exports - Treat warnings as reviewer prompts, not noise
Don't
- Read a clean run as a review
- Add a hallucinated name to package.json to silence it
- Allow-list props to pass a PR; change the contract instead
- Expect it to see accessibility, logic or forbidden combinations
Appendix B
Put the checklist where the pull request is.
GitHub reads pull request templates from the default branch. Several templates go in a PULL_REQUEST_TEMPLATE folder and are chosen with a template= query parameter [S08].
.github/PULL_REQUEST_TEMPLATE/ai-assisted.md (excerpt)
## What an AI tool wrote (C01)
- Tool and model:
- Task or prompt given (link or paste):
- Files or parts written by the tool:
## Machine checks (C05)
```text
typecheck:
lint (including token rules, Field Guide 03):
tests:
check-ai-diff.mjs:
```
## Dependencies and imports (C07 to C13)
- [ ] No new dependency, or each new one is listed here with its `npm view` result
- [ ] I did not install anything to find out whether it exists.
## Tests (C34 to C38)
- [ ] Every new test fails when the behaviour it names breaks. I broke it once to see.
- [ ] No test was skipped, deleted or loosened to make this pass.Open it with ?quick_pull=1&template=ai-assisted.md on the compare URL, or copy it to .github/pull_request_template.md to make it the default.
| Where it runs | What it covers | Items |
|---|---|---|
| Author, before opening | Disclosure, ran it, scope | C01 to C04 |
| CI | Typecheck, lint, tests, check-ai-diff.mjs | C05, C07, C11 to C17, C19, C20, C28, C29, C34 to C36 |
| Reviewer, reading | Packages, lockfile, patterns, security, test quality | C08 to C10, C18, C21, C30 to C33, C37, C38 |
| Reviewer, using it | Keyboard, names, announcements, edge cases | C22 to C27 |
Keep with the pull request
Leave a review record.
For pull requests where generated code is a large share of the diff. It shows what was verified by whom, so the next incident review does not start from nothing.
Sources / maintenance
Keep the guide current.
Sources checked 24 September 2026. Research and vendor figures are cited as their authors report them. The fixture package name @acme/ui-hooks returned 404 from the npm registry on that date; do not register or install it.