Exploited, not suspected: what DAST false positives really cost.

A long report is a transfer, not a result

Most scanners hand you a list that mixes what is real with what might be. The list feels thorough. Then somebody has to sort it. DAST false positives are not a nuisance at the edge of that work. Sorting them is the work, and it lands on your team after the vendor has been paid.

That sorting has a cost nobody invoices you for, so it rarely makes it into the comparison spreadsheet when a tool is chosen. This post is about putting it there, and about the one reporting rule that changes the sum.

Why DAST false positives happen at all

It is not carelessness. Proving an item takes a running application, a working session, and a request that has to succeed without breaking anything along the way. Matching a pattern takes none of that, so it is faster, cheaper and easier to run across a large estate. A tool is also blamed more for missing something than for listing too much, so most of them lean towards listing.

For breadth, that is a reasonable design, and we are not arguing against it. It simply moves the cost of being sure from the vendor to you, and that cost deserves a line of its own when you compare tools.

The arithmetic of triage

Triage is three costs, not one.

Time to disprove. Reading the item, reproducing it, checking whether the input is reachable, finding the code. An obvious false positive dies in minutes. A plausible one can take an afternoon, and the plausible ones are the expensive ones precisely because they look like everything else on the list.

Interruption. The person who can disprove an item fastest is usually the engineer who owns the code, which means pulling them off whatever they were building. The cost is the context switch as much as the hour.

Trust. The largest cost, and the one no timesheet records. A team that works through a long list and finds mostly nothing learns that the list is not worth reading. The next report gets skimmed, and the one real item inside it waits behind the noise.

You can put a rough number on the first two for your own team without anyone's statistics. Take the items in your last report, multiply by the average time it took to close one out, and multiply again by the hourly cost of whoever did it. Then add, separately, the real items that sat for weeks because nobody trusted the list. That second line is usually the one that matters.

What "confirmed" has to mean

We have written up the anatomy of a finding before, in what belongs in a finding: the request, the response, a screenshot and the line of code. Here the question is narrower. Which of those parts proves an item is real, and which proves it is fixable?

The request and the response together prove that something happened. The screenshot proves it to the people who are not going to read a raw response. The line of code proves where the change belongs. An item with a rule name, a location and a severity colour, but no captured exchange, proves only that a pattern matched.

A useful test works on any line in any report: could someone outside the vendor reproduce it from the page alone, in ten minutes? If yes, it is a finding. If not, it is a lead, however confident the colour looks. "Potential" is a polite word for "now yours to prove".

Confidence is not severity

Severity answers how bad something would be if it is real. Confidence answers how sure anyone is that it is real. Put them on two axes and each corner asks for a different decision.

Most tools sort by a single colour that blends the two. A critical that nobody has proved sits above a medium that was exploited, and the team works the wrong one first. We keep severity and confidence as separate fields, with CWE and OWASP references on each finding, so a list can be sorted by either and nobody has to guess which one a colour means.

Confirmed-only reporting, and what it costs us

The rule is short: if we did not break it, we do not report it. Each finding on the list your team acts on was exploited, and carries the request, the response, a screenshot where a person can see the result, and, when we can read your code, the line that allowed it. Anything we suspected but could not confirm stays out of that list.

The trade is real and you should know it before you buy. Some of what a tool suspects and cannot confirm is genuinely there: a bug behind a flow the tool could not complete, a race it could not time, a condition that only holds with data it did not have. Confirmed-only reporting will not show you those. It gives up some recall to buy a list that is worth reading.

We think that is the right trade for the list a team works from each week, because an unread list finds nothing in practice, however long it is. It is the wrong trade if what you need is the widest possible inventory of suspected weaknesses for an audit. In that case you want breadth, and you should budget honestly for the triage that comes with it.

How to judge any vendor's findings

You do not need to take our word for any of this. Five questions work on any report from any vendor, ourselves included, and they work best on a real run against your own application rather than on a sample from the vendor's demo app, which only shows you their best day.

1. Can I reproduce it from the page? The request, the response, and which account sent it. If any of those is missing, someone on your team will have to rebuild it.

2. Was it exploited or inferred? Ask the vendor to point at the evidence that it happened, not the rule that matched. Both are useful. Only one is a finding.

3. Are severity and confidence separate? If the report has one colour, ask which of the two it measures. If the answer is "both", the sort order is guessing.

4. Where in the code? A file and line, or at least the handler, turns a finding into a pull request. Without it, the fix starts with a search.

5. How is a fix proven? Re-running the exact exploit against the new code is proof. A fresh scan that no longer mentions the item is not, because a scan that comes back empty may simply have taken a different path. That re-run is how we close the loop on our own findings.

Honest limitation

Confirmed-only reporting lowers recall on purpose, as described above. It also depends on what a test is allowed to do. Confirmation needs permission to exploit, so on an environment where that permission is narrow, fewer items can be confirmed. The list will be shorter for that reason, not because the application is cleaner, and a good report should say which it was.

We have also not published a detection rate, and we will not make one up. When our benchmark publishes, the misses go in with it. Why we publish what we miss →

Monday-morning actions

1. Pull your last report. From any tool or tester. Count the items that carry all four parts: request, response, screenshot where relevant, and line of code.

2. Price the rest. Use the sum above on the items that did not, and write the figure next to the licence cost.

3. Send them back. Return the incomplete items to whoever produced them and ask for the request and response, or a plain statement that they were not confirmed.

4. Split your backlog. Add a confidence field beside severity and re-sort. Notice which items move.

5. Ask the five questions. On the next vendor call, ask them about a finding from your own application, not one from a slide.