Gating Token Pull Requests with Playwright Snapshots

Part of Visual Regression & Token Drift. A snapshot suite that exists is not a gate; a snapshot suite that blocks a merge is. This guide covers the wiring that makes the second one survivable: what triggers it, what the failure means, and how an intentional change gets through without disabling anything.

Two states a snapshot difference can be in A snapshot difference is shown branching into an intentional change, which is resolved by committing a new baseline, and an unintentional one, which is resolved by fixing the code. Snapshot check fails Intentional the token change was the point reviewer sees old and new side by side resolution: commit the new baseline Unintentional a component changed as a side effect no token in the diff explains it resolution: fix the code, not the image A gate that cannot distinguish these two produces the same pressure in both cases — to update the baseline and move on.
The entire design of the gate follows from making the left branch cheap and the right branch obvious.

Prerequisites

  • A working component snapshot suite, as built in snapshotting component screenshots.
  • Baselines committed to the repository, so a new baseline appears in the pull request diff.
  • A CI provider that supports required status checks and path filters.
  • Agreement on who approves a baseline change; in most teams this is the design system maintainers plus the pull request author.

Step-by-Step Implementation

Step 1 — Trigger on the paths that can change pixels

Intent: the check should not run on a documentation edit, and it must run on every change that can alter rendering.

# .github/workflows/snapshots.yml
name: Component snapshots
on:
  pull_request:
    paths:
      - 'tokens/**'
      - 'packages/components/**'
      - 'packages/styles/**'
      - 'tests/__screenshots__/**'
      - '.github/workflows/snapshots.yml'
concurrency:
  group: snapshots-${{ github.head_ref }}
  cancel-in-progress: true

Why this works: including the baselines directory in the trigger is the non-obvious entry. A pull request that only updates baselines must re-run the check, otherwise a baseline can be committed that does not match what the suite produces, and the next unrelated pull request fails for no reason anyone can trace.

Step 2 — Run in a container, not on the bare runner

Intent: font rendering is the largest source of cross-environment difference, and a container fixes it.

jobs:
  snapshots:
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.49.0-noble
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run build
      - name: Compute affected components
        id: affected
        run: echo "list=$(node scripts/affected.mjs)" >> "$GITHUB_OUTPUT"
      - name: Snapshot
        env:
          AFFECTED: ${{ steps.affected.outputs.list }}
        run: npx playwright test --project=components

Why this works: pinning the container to the same tag as the Playwright dependency means the browser, the operating system libraries and the installed fonts are all fixed together. Upgrading becomes a deliberate act that regenerates baselines in one pull request, rather than a surprise the day the runner image changes.

Step 3 — Publish the diff as an artefact and a comment

Intent: a reviewer should be able to see the difference without downloading anything.

      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: snapshot-diff
          path: test-results/
          retention-days: 14
      - name: Comment with the token context
        if: failure()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const changes = JSON.parse(fs.readFileSync('dist/changed-tokens-detail.json', 'utf8'));
            const lines = changes.map(c => `- \`${c.name}\`: ${c.from} → ${c.to}`).join('\n');
            await github.rest.issues.createComment({
              ...context.repo, issue_number: context.issue.number,
              body: `**Snapshots differ.** Token values changed in this PR:\n\n${lines || '_none_'}\n\n` +
                    (lines ? 'If the visual change matches the token change, update the baselines.' :
                             '**No token values changed** — this difference came from component or CSS code.'),
            });

Why this works: the two-branch message does the classification for the reviewer. When no token changed, the comment says so explicitly, which turns the most dangerous case — an unintended side effect — into the loudest one rather than the quietest.

Where a snapshot gate sits among the other checks Five checks ordered by runtime: schema, references, lint, contrast and snapshots, with the snapshot check last and marked as the only one requiring human judgement. schema ~1s references ~2s lint ~15s contrast ~40s snapshots ~60s Only the last one can fail for a good reason The first four failures are always defects. A snapshot failure may be the intended outcome, which is why it needs an approval path.
Ordering by runtime is conventional; the important property here is that the only judgement-requiring check runs last, after everything mechanical has passed.

Step 4 — Make baseline updates a reviewable commit

Intent: the approval mechanism should be the ordinary code review, not a separate tool.

# On the branch, after confirming the change is intended:
AFFECTED="$(node scripts/affected.mjs)" npx playwright test --update-snapshots
git add tests/__screenshots__
git commit -m "chore(snapshots): update baselines for spacing scale change

The inset scale moved from 16px to 12px at the md step. Affected:
button, card, dialog, form-field."

Why this works: the commit message records why the baseline changed, which is the piece of context that is impossible to recover later from an image diff. Six months on, “why is this button’s padding different from the design file” is answerable by git log on the baseline image.

Step 5 — Provide an escape hatch that leaves a trace

Intent: occasionally a change must ship before its snapshots can be regenerated. That should be possible and visible.

      - name: Snapshot
        id: snap
        continue-on-error: ${{ contains(github.event.pull_request.labels.*.name, 'skip-snapshots') }}
        run: npx playwright test --project=components
      - name: Record the skip
        if: steps.snap.outcome == 'failure'
        run: |
          echo "::warning::Snapshot failures were bypassed via the skip-snapshots label."
          echo "PR #${{ github.event.number }} bypassed snapshots on $(date -I)" >> .skip-log

Why this works: a label is visible on the pull request, searchable afterwards, and requires a deliberate act. Compare that with the alternative escape hatches — commenting out a test, or marking the check non-required — both of which are invisible a week later. A monthly review of the label’s usage tells you whether the gate is working or being routed around.

Step 6 — Keep the check required, but only where it can run

Intent: a required check that cannot run on some pull requests blocks them forever.

  snapshots-gate:
    needs: [snapshots]
    if: always()
    runs-on: ubuntu-latest
    steps:
      - name: Conclude
        run: |
          result="${{ needs.snapshots.result }}"
          if [ "$result" = "failure" ]; then exit 1; fi
          echo "Snapshots: $result"

Why this works: a small always-running job that concludes on behalf of a path-filtered one is the standard pattern for making a conditional check requirable. Without it, a pull request that touches no relevant path never reports the required check, and the merge button stays disabled with no explanation.

Verification

Confirm all four behaviours before declaring the gate live: a token change that alters rendering fails the check; committing regenerated baselines makes it pass; a pull request touching only documentation reports the check as skipped and remains mergeable; and applying the skip label allows a merge while emitting a warning.

The third is the one that is usually missed and the one that causes the most disruption, because it manifests as “the merge button is greyed out and no check is failing” — a state that consumes a surprising amount of collective time to diagnose the first time it happens.

Signals that a snapshot gate is being routed around Three warning indicators — rising skip label usage, baselines updated in commits with no token change, and re-runs without code changes — each with what it implies about the gate. Skip label usage rising more than once a month across different authors implies: the suite is too slow or too noisy to be worth it Blind baseline updates baselines changed in commits with no token change implies: real regressions are being approved away Re-runs without changes the same commit re-run until it passes implies: flakiness remains fix determinism first
All three are measurable from repository metadata, and all three appear well before anyone proposes removing the gate.

Troubleshooting

Symptom Likely cause Fix
The merge button is disabled with no failing check A required check that was skipped by a path filter Add the always-running conclusion job from step 6
Baselines pass locally and fail in CI Baselines were generated outside the container Generate baselines only in CI, or run the same container image locally
The check re-runs on every push and takes minutes No concurrency group, so superseded runs continue Add concurrency with cancel-in-progress
A baseline commit itself fails the check The baselines path is not in the trigger filter Add tests/__screenshots__/** to the paths list
Diff artefacts expire before review Default retention is short for large artefacts Set an explicit retention period, and keep the comment as the durable record

The Difference Between a Gate and a Suite

It is worth separating two things that are easy to conflate. A test suite tells you whether something changed. A gate decides whether that change may proceed. The same Playwright run can serve both roles, and the design decisions are different for each.

As a suite, more coverage is straightforwardly better and a slow run is merely inconvenient. As a gate, coverage has a cost paid by every contributor on every change, and a slow run is a tax on the whole team’s throughput. The moment a suite becomes a gate, its runtime becomes a shared resource that needs managing.

The second difference is about failure semantics. A suite may reasonably report “this looks different, take a look”. A gate has to answer “may this merge”, and every failure needs a defined resolution path. That is why the escape hatch matters: without one, the only resolutions available are “regenerate the baseline” — which may be wrong — and “argue with the pipeline”, which is worse.

The practical consequence is that a suite can grow organically and a gate cannot. Keep the gate to the components where a regression would be genuinely expensive, and run the broader suite nightly where its failures are informational. Teams that promote their whole suite to a gate in one step almost always demote it again within a quarter.

Migration Note

Introduce the gate in reporting mode for at least two weeks. The job runs, comments on differences, and never blocks. That period produces the two numbers that decide whether the gate is viable: how often it fires, and what proportion of those firings were genuine regressions. A suite firing weekly with a high genuine rate is ready to block; one firing daily with a low rate needs its determinism fixed first.

When it does become required, announce the change with the escape-hatch label documented in the same message. Teams accept a new blocking check far more readily when the way past it is stated up front, and the label usage is a better signal about the gate’s health than any complaint would be.