A validated security sweep for production web apps, without a scanner bill

The situation

Small teams ship web apps without a security budget. Commercial scanners charge per target or per seat, and their reports mix real issues with long lists of findings that do not apply. The usual outcome is a report that goes unread.

We wanted something we could run before every release on the apps we look after: our own site and the client web apps we maintain. It had to be cheap enough to run often and strict enough that a finding in the report always means something.

What we did

A collector that does the same thing every time

The tools run in a disposable sandbox container that is created for the run and removed after it. A plain shell collector clones the repository, runs the scanners against the code and the live site, and writes the raw output into a dated run directory:

• gitleaks for committed secrets,
• trivy for known vulnerabilities in dependencies,
• semgrep for risky code patterns,
• an HTTP probe that records the response headers,
• exposure probes for files that should never be public, such as .env or .git/config,
• nuclei for known web issues,
• a screenshot of the homepage, to confirm the site rendered and shows no error page.

An earlier version let a model decide which tools to run. It was unreliable: runs skipped tools and produced different coverage from one week to the next. Splitting the work fixed that. The collector is deterministic, and judgment comes only after the data exists.

Triage that has to prove each finding

The raw output then goes through an AI triage step, validated by a human, with one rule: a finding is reported only with a file and line or a reproduced request. Scanner hits that fail the test are written down as discarded, with the reason. Typical discards are test keys in test files, a deliberate dummy password hash, and a dependency CVE in code the app never calls.

The review also covers what scanners miss. For Next.js apps that means every server action (is the user checked before any data access, and is the record scoped to them), the route protection file, payment webhooks (signature verified on the raw body, prices taken from the database) and any place the server fetches a URL it was given.

A report that credits what works

Each run produces a findings file and a short report: counts, the findings, the controls we verified as working, and what the run could not cover. The positive controls matter. A report that lists only problems gives no way to tell a careful app from one that was never checked.

Results go to a team channel on every run, including clean ones, so a missing message means the job broke. Findings go to the owner of each app. This page describes only findings that have been fixed.

Results

Between the two runs, the client app's runtime database connection moved from encrypted to encrypted and verified: it now checks the server certificate against a pinned certificate authority instead of trusting whatever answers.

Limits we state in every report

• nuclei returns little behind Cloudflare and Vercel, which throttle it. We record that instead of treating an empty result as clean.
• The sweep does not test signed-in flows. Access control is checked by reading the source.