CSP policies regress silently. That’s the whole problem.

Someone adds 'unsafe-inline' to script-src to unblock a third-party widget on a deadline. It ships. The policy that took a quarter to tighten is now decorative, and nothing anywhere fails. No test breaks, no alert fires, no reviewer catches it because the diff is one token long and the PR is about a marketing pixel. You find out months later, from an XSS report, if you find out at all.

Every other class of regression we’ve learned to catch mechanically. Type errors, broken tests, dependency CVEs, coverage drops — all of them block a merge. Security headers sit outside that loop almost everywhere, checked once during a pentest and then left to drift.

This post is about closing that gap: what to run in CI, what to gate on, and a couple of design details that decide whether the check survives contact with a real team or gets disabled within a month.

I’ll use csp-toolkit — my Python CSP analyzer — because gating is what I built into v0.8. The reasoning applies to whatever you run.

The smallest useful version

If your CSP lives in config, check it before it ships:

csp-toolkit analyze --fail-on high -f config/csp.txt

That exits non-zero if any finding is CRITICAL or HIGH. Drop it in a pull request job and you’ve covered the common case, which is someone loosening a directive without realizing what it costs.

If your CSP is assembled at runtime — middleware, a CDN edge worker, a framework plugin — checking a config file proves nothing about what browsers receive. Check the deployed header instead:

csp-toolkit fetch https://staging.example.com --fail-on high

Same gate, live target. Run it against staging after deploy, before promotion.

That distinction matters more than it sounds. A policy that’s correct in source and mangled by a proxy is a policy that’s wrong in production, and only the second check catches it.

Why the gate exits 3

Here’s the detail I got wrong on the first pass, and it’s the kind of thing that’s invisible until it bites.

The obvious exit code for “gate failed” is 1, or 2. I reached for 2. It’s wrong, and a test caught it.

Click — like argparse, and like a lot of Unix tooling — already uses exit code 2 for usage errors. So with the gate on 2:

csp-toolkit analyze "$CSP" --fail-on hgih   # typo → exit 2
csp-toolkit analyze "$CSP" --fail-on high   # real regression → exit 2

Identical status. A CI wrapper cannot distinguish them, and the failure modes are opposites. One means your policy got worse and needs a developer. The other means your check is broken and has never analyzed anything, and needs a config fix.

The second is the dangerous one, because it looks like the check is working. A red X appears on every build. Someone assumes the CSP is failing, adds an ignore, and now you have a security check that is permanently green and permanently blind.

So the gate exits 3:

CodeMeaning
0Passed, or no gate flag was given
1Runtime error — network failure, unreadable file
2Usage error — bad flag, bad value
3Policy gate violated

Three states that need three different human responses, so they get three different codes. The wrapper treats 3 as “report the findings and fail the build” and anything else non-zero as “the tool itself is broken, surface it loudly.”

The general lesson isn’t about CSP: if your tool introduces a “check failed” exit status, make sure it doesn’t collide with the exit status your CLI framework already uses for “you called me wrong.” A security check that can’t distinguish its own misconfiguration from a real finding is worse than no check, because it manufactures confidence.

What to actually gate on

The gate flags are the easy part. Choosing the threshold is where these checks live or die.

Start with severity, not grade. csp-toolkit scores policies A+ to F, and gating on --min-grade B is tempting because it’s one number. Don’t lead with it. A grade is a composite that moves for reasons unrelated to the change in front of you — a developer touching a font directive can drop the grade through no fault of their own, and they now have to reverse-engineer a score to get their PR merged. --fail-on high names an actual condition, and the failure message points at the specific finding.

Use a grade floor later, as a looser secondary signal, once the severity gate has been clean for a while:

csp-toolkit analyze --fail-on high --min-grade C -f config/csp.txt

Don’t gate on Report-Only. A Content-Security-Policy-Report-Only header doesn’t block anything — the browser reports and moves on. A weakness in a policy with no enforcement effect isn’t exploitable, and failing a build over one teaches people the check is noise. csp-toolkit reports findings on Report-Only policies and never gates on them; if you’re rolling your own, make the same carve-out.

Ratchet, don’t big-bang. If your current policy grades D, --fail-on high will fail on day one and stay failing, and the check will be removed by Friday. Start at --fail-on critical, get to green, fix the highs deliberately, then tighten. A gate that’s been green for a month is a gate people trust when it goes red.

Getting findings in front of the developer

An exit code tells you something is wrong. It doesn’t tell you which directive, and log output is where security findings go to be ignored.

SARIF fixes that. csp-toolkit emits SARIF 2.1.0, which GitHub renders as inline pull request annotations:

csp-toolkit analyze -f config/csp.txt --fail-on high -o sarif --output csp.sarif

Wrapped up as an Action:

name: CSP Check
on: [pull_request]

permissions:
  contents: read
  security-events: write   # required for SARIF upload

jobs:
  csp:
    runs-on: ubuntu-latest
    steps:
      - uses: sampsonc/csp_toolkit@v1
        with:
          url: https://staging.example.com
          fail-on: high
          min-grade: B

Or against a file, when you’d rather catch it before deploy:

      - uses: sampsonc/csp_toolkit@v1
        with:
          policy-file: config/csp.txt
          fail-on: critical

The finding lands next to the diff that caused it, which is the difference between a developer fixing it now and a ticket nobody picks up.

One implementation note if you build something similar: make the SARIF upload happen even when the gate fails. The natural shape — run the tool, exit on failure — skips the upload on exactly the runs where the findings matter most. The Action captures the status, uploads the report, and fails afterward.

What this doesn’t do

Worth being clear about the limits.

A CSP grade is not a measure of whether you’re exploitable. A policy can grade A and still be bypassable through a JSONP endpoint on an allowlisted domain, and csp-toolkit’s bypass database exists precisely because allowlists are where good-looking policies go wrong. Gating catches regressions; it doesn’t validate that your policy is sound in the first place. That’s still a review problem.

It also won’t catch a policy that’s correct in CI and different in production — different edge config, a proxy rewriting headers, a CDN injecting its own. Checking the deployed header rather than the source file narrows this, but only for the environment you actually check.

And a check that fires on every PR is only useful if it’s fast and quiet. If it flaps, people route around it.

The setup I’d recommend

Two jobs, different jobs:

  1. On every pull request — check the policy file with --fail-on high. Fast, no network, blocks the merge.
  2. After deploying to staging — check the live header with fetch --fail-on high --fail-on-missing-csp. Catches the delivery path, including the case where the header vanishes entirely.

The second flag is worth having. A policy that silently stops being sent is the most complete regression available, and a source-file check will happily pass while it happens.

csp-toolkit is on PyPI and GitHub, MIT licensed, with docs here. The gating flags and the Action landed in v0.8.0; v0.8.1 fixed a Report-Only gating inconsistency found while writing this post.

See also: