Our false-positive rate, in public
False-positive floods are the standing complaint about every tool in this category, and not one of them publishes its rate. Here is ours, with the corpus it was measured against, the date, the commit, and the command that reproduces it. Recall stands next to precision on every row — a tool that reports nothing scores a perfect precision, so a precision shown on its own is not evidence of anything.
Measured 2026-08-11 against commit f597d9577f81 · 9 repositories · 24 planted findings
What this number is not
- · 9 repositories are not 6,000. This is a corpus we wrote: 2 that must report nothing at all and 7 carrying at least one deliberately planted flaw. It is a regression gate that a rule has to clear before it ships. It is not a measurement of how the scanner behaves across the whole of GitHub, and nobody should read it as one.
- · We wrote the answer key. The corpus and the detectors come from the same hand, and marking your own homework is worth exactly as much as the homework being readable. So it is: every fixture and every answer key is a file in the repository, listed below, rather than a description of one. Nothing here is measured against code you cannot see.
- · The deep-scan rows measure the pipeline, not the model. This run replayed a fixed set of model answers, so those rows say that anchoring, verification, de-duplication and assembly did their job — they do not say the model judged well. Re-running with
EVAL=livemeasures the model instead, and that number will be lower. - · A potential finding is not a false positive. When static analysis can show a check is missing but cannot show it is exploitable, we label the finding potential and say so in the report. It has claimed nothing it did not deliver. What counts against us here is a finding on a route that has no problem at all — which is what the silent repositories in the corpus exist to catch.
The four deterministic checks
No model is involved in these. They run over your code and produce the same answer every time, so this table is a measurement of the checks themselves, end to end.
| Check | Precision | Recall |
|---|---|---|
| C1 — Row Level Security | 1.0002/2 | 1.0002/2 |
| C2 — exposed secrets | 1.0001/1 | 1.0001/1 |
| C3 — broken access control | 1.0002/2 | 1.0002/2 |
| C4 — public storage buckets | 1.0001/1 | 1.0001/1 |
The AI Deep Scan
Findings the research pass formulates itself, plus the five vetted agent categories. Every one of them ships as potential by design, and every one has to quote a line of your code that exists character-for-character or it is dropped before you ever see it.
| Check | Precision | Recall |
|---|---|---|
| A2 — authenticated but unscoped | 1.0002/2 | 1.0002/2 |
| A3 — unverified JWT | 1.0001/1 | 1.0001/1 |
| Open research — no fixed category | 1.00015/15 | 1.00015/15 |
Same input, same output
Scan the same commit twice and the four rule checks return the same findings, in the same order, every time. No sampling, no temperature, no model — they are a SQL parser, a set of regular expressions, a route-handler heuristic and a bucket-config reader, run over files read once. If you get a different answer on the second run, your code changed or we have a bug, and both are things you can hold us to.
The grade is the same promise. It is five comparisons — critical > 0 → F, high > 0 → D, medium > 1 → C, medium = 1 → B, else A — and nothing else. There is no weighted score, because a formula nobody can recompute is an assertion, not a grade, and a number that drifts between 79 and 94 on the same codebase is worse than no number at all.
The AI Deep Scan is not deterministic, and no wording here will make it so. That is why it is a separate section of the report, why it is excluded from the count, the grade and the monitoring diff, and why every one of its findings ships as potential with a verbatim quote of the line it rests on. Two runs may phrase the same issue differently or surface a different subset. The half of the product that alerts you at 3am is the deterministic half, on purpose.
How a finding is scored
- · True positive — the corpus plants this finding, at this location, under this check.
- · False negative — the corpus plants it and the scan did not report it.
- · False positive — the scan reported it in a repository the corpus declares silent, or on a path the corpus pins as clean.
- · Unclassified — reported somewhere the corpus takes no position. Counted separately and never folded into either rate. This run had 0.
Precision is tp / (tp + fp), recall is tp / (tp + fn). Rates are truncated, never rounded — 0.94 is printed as 0.940 and 0.9995 is printed as 0.999.
Reproduce it
The corpus lives in test/fixtures/repos, one directory per repository, each with an expected.json stating what is planted there and what must never be reported. The vulnerable repositories are written by us; no customer code is in any of them.
npx tsx scripts/precision-report.ts # writes src/lib/precision.json npm test -- precision # fails if the number is stale
The number is pinned to the code by content digest rather than by a timestamp, so a fresh clone verifies it identically:
- detectors
- b7e86949cf18ff1ec2cb4f77a984ca169b16391ae9acde7f8f991270f4436415
- corpus
- 120cf65bd95f1811eeb5aad651c919d7d361672a2b1aa28aa5d698e4b89d7213
A test fails the build when either digest stops matching the files on disk. That is what stops this page from stating, three months from now, a rate that belongs to code nobody is running.
When it gets worse
If a new rule pushes precision down, the number goes down. It does not get held back, and the rule does not get hidden. The entire worth of this page is that you can believe the figure on it, which means it has to be allowed to be unflattering. If you find a false positive we have not counted, tell us — a corpus repository that breaks a rule is the most useful thing anyone can send.