How to review every pull request as commits land

Human review arrives in a batch, late, after the author has moved on. An agent can read each push as it lands instead. The risk is noise: a naive bot re-reviews the entire diff on every commit, and the team mutes it within a week.

Code review is built for this. It reviews only the commits it has not seen yet, it never re-comments a line it already reviewed, and a second pass decides what is worth showing you before anything posts.

Turn it on

Code review is one organization-wide setting, off by default. Enable it on the Settings tab of /reviews in the dashboard. No YAML required: every pull request in the organization is then reviewed by the built-in pipeline.

That pipeline is two agents. The bugs reviewer reads the range on the frontier model, briefed to find defects: logic errors, unhandled edge cases, error handling, concurrency, security holes, regressions in behavior callers depend on, and migration hazards. It reads the surrounding code, not just the diff, because most real findings depend on a caller or a guard the diff does not show.

Then the gatekeeper reads every finding the reviewer proposed and decides what posts. It is an independent second read on the frontier model, not a second reviewer: it checks each claim against the code and drops the ones that do not hold, are already handled elsewhere, or are style opinions. That pass is what keeps an eager reviewer from becoming noise.

When priya-shah opens "Expire pending trades on account deactivation" and pushes three more commits over the afternoon, the pipeline runs four times, each time reading only what that push added. The third push drops a rest-period check and the comment lands on that line, with no repetition of the first review.

Scope it with a pipeline file

A file is only needed to change something. Commit agents/code_review.yaml to scope which pull requests get reviewed:

ellipsis:
  version: v1
  kind: code_review
  name: Backend review

pull_requests:
  repositories: [splitshift-api, splitshift-web]
  base: [main, release/*]
  draft: false
  for:
    users: true
    bots: false

kind: code_review is what makes this a pipeline rather than an agent config, and it is required. Everything else is optional: this file changes the watch set and inherits the three reviewers, the gatekeeper, and the budget from the platform.

A committed file makes its filters authoritative. Scoping review to splitshift-api and splitshift-web means "and nothing else", so a pull request in splitshift-mobile gets no review at all rather than falling back to the unscoped default. That is the point of scoping: Ellipsis never reviews, or bills you for, something your file excluded.

for.bots: false is the default, so Dependabot pull requests never trigger a review.

Add a reviewer without losing the built-in ones

review: normally replaces the built-in reviewer. include_default_reviewers: true adds to it instead, which is what you want when the built-in review is fine and you need one more pass alongside it:

ellipsis:
  version: v1
  kind: code_review
  name: Backend review with a migration specialist

pull_requests:
  repositories: [splitshift-api]

include_default_reviewers: true

review:
  - name: migration-safety
    claude:
      model: claude-opus-4-8
      system: |
        You review database migrations for production safety.

        Check for locking behavior that blocks writes on a large table,
        missing backfills for new non-null columns, missing indexes on
        new foreign keys, and rollout ordering that breaks if the
        migration and the code deploy in the wrong order.

        Comment on the specific line with the specific change you want.
        A migration that is safe as written gets a one-line
        confirmation, not invented concerns.
    pull_requests:
      paths: ["**/migrations/**"]

budget:
  run: 12.00

Two reviewers now run: the built-in bugs plus this one. Without the flag, review: would have left you with only migration-safety, and keeping the built-in review would have meant copying its prompt into your file, where it stops tracking improvements to ours.

The built-in runs first, then yours.

Per-reviewer filters

migration-safety carries its own pull_requests: block, so it only runs when a migration is actually in the diff. A reviewer that filters out costs nothing: no sandbox starts and no model call is made, and its share of the run budget goes to the reviewers that did run.

A reviewer's block takes paths, base, and head only. repositories belongs to the pipeline, which owns the watch set. When dani-okoro adds a non-null expired_at column, migration-safety runs alongside the built-in reviewer and the review lands before anyone else looks:

migrations/0042_expired_at.py:14 — adding a NOT NULL column with a
default rewrites the whole table under an ACCESS EXCLUSIVE lock.
On trade_requests (~50M rows) that blocks writes for minutes.

Split it: add the column nullable, backfill in batches, then set
NOT NULL in a follow-up migration once the backfill completes.

On a pull request that touches no migrations, that reviewer never starts.

Replace the reviewers entirely

Leave include_default_reviewers off and review: replaces the built-in, which is what you want when your own prompts are the review:

ellipsis:
  version: v1
  kind: code_review
  name: House review

pull_requests:
  repositories: [splitshift-api]

review:
  - name: house-rules
    claude:
      model: claude-haiku-4-5-20251001
      system: |
        Review this diff against the conventions in CONTRIBUTING.md
        and the service patterns in docs/architecture.md. Comment only
        where the diff departs from a documented convention, and cite
        the document. Skip anything a linter catches.

budget:
  run: 4.00

A pipeline may declare at most 8 reviewers, counting the built-in when include_default_reviewers is on. Asking for more is a validation error rather than a silent trim. Reviewers run in parallel, so several cost latency only once.

Turn the gatekeeper off

The gatekeeper suppresses findings it judges to be noise, wrong, or already covered elsewhere. Set filter: [] to skip it and post everything the reviewers found:

filter: []

This trades precision for completeness, and the trade is real. Reviewers that phrase the same defect differently produce a comment each, so expect the same issue raised more than once. Mechanical deduplication catches only identical findings. The posted review says plainly that no gatekeeper ran.

Turn it off when you would rather triage a noisy review yourself than risk a real defect being suppressed. Leave it on otherwise.

Budgets

budget.run caps one pass over one pull request, across every reviewer and the gatekeeper together. Reviewers that declare no budget of their own split what is left equally:

budget:
  run: 10.00
  day: 60.00
  week: 200.00

day and week are this pipeline's trailing spend, checked before a review starts. A pipeline over its daily cap reviews nothing until the window rolls, which is the guard against a push storm turning into a bill.

Reviewers fan out, so one push is several sessions. A cheap model is the right default for a reviewer that runs on every push; save the expensive one for the gatekeeper, which runs once per review, and for specialists that fire rarely.

Cross-repository context

pull_requests.repositories is the watch set. sandbox.repositories is what gets cloned. They are independent, so a reviewer can read a pull request in one repository with another checked out for context:

ellipsis:
  version: v1
  kind: code_review
  name: API reviewer with web context

pull_requests:
  repositories: [splitshift-api]

sandbox:
  repositories:
    - name: splitshift-web
    - name: splitshift-shared

Reviews trigger only on splitshift-api pull requests, and splitshift-web plus splitshift-shared are on disk while reviewing. The triggering repository is always cloned, so it does not need listing.

This is how a reviewer judges a change against callers it does not contain: an API pull request that changes a response shape can be read against the web client that consumes it. It needs no second pipeline file.

One pipeline per pull request

At most one pipeline may review a given pull request. Two enabled files whose watch sets overlap is a configuration error, reported on the newer file, and the older one keeps running. Narrow one file's repositories, or disable one.

So a specialist reviewer is a reviewer inside your pipeline, not a second file. That is what per-reviewer pull_requests: filters are for.

Next

Keep descriptions current on the same pushes with How to keep pull request descriptions up to date, review one narrow class of change with a dedicated agent in How to review database migrations before they merge, or see Budgets for what a cap does when it is reached.