Penetration testing evidence: what counts, and what an auditor can check

A pentest finding is only as good as the record behind it. In Arxiis, a finding is labelled proven only when its evidence binds to a real record in a tamper-evident execution ledger. Everything else is labelled needs review. This post covers what counts as penetration testing evidence for the engineer and for the auditor. It also covers what a ledger record holds, and how to retain the evidence and hand it over.

Two readers, two questions

Evidence has two audiences, and they ask different questions.

The engineer asks: is this real, and where exactly is it? They need the request that was sent, the response that came back, and enough context to reproduce the behaviour in their own environment. A severity rating and a paragraph of prose do not let them start work.

The auditor asks: was this test authorized, and was it done the way the report says? Does each conclusion follow from something that was observed? They care less about the bug than about the chain from permission to conclusion.

A report that serves only one of them fails the other. A screenshot as proof of exploit satisfies neither for long. The engineer cannot replay a screenshot, and the auditor cannot tell when it was taken or whether it was edited.

What the guidance counts as evidence

The PCI Security Standards Council's Penetration Testing Guidance, an information supplement last revised in September 2017, gives a definition worth borrowing. Evidence is all information that supports the tester's conclusions. Its examples include screenshots, raw tool output and recordings. It recommends a systematic process to collect, handle and store evidence securely.

The same supplement recommends that retention and destruction procedures be documented for all parties before testing starts. It says that, at the time, PCI DSS set no requirement for retaining evidence the tester collects. It calls retention a recommended best practice, with evidence available on request as the rules of engagement define. It was written against an earlier version of PCI DSS, so check current pentest evidence requirements with your assessor.

Two gaps in that definition matter for automated testing. Raw tool output is only evidence if you can show which run produced it and that it has not changed since. And "anything that supports the conclusion" leaves the link between a finding and its evidence to the tester's judgement. We built the ledger to close both gaps.

What a ledger record holds

Every command the crew runs is written to an append-only execution ledger. Each record carries five things:

• a sequence number
• a timestamp
• the command that was run
• the observed response
• a hash that includes the previous record's hash

Escalation approvals and declines are written to the ledger too, with the operator's identity. So the record of who allowed what sits in the same chain as the record of what was done.

Because each hash includes the one before it, editing any record breaks every hash after it. That makes the ledger tamper-evident, and only tamper-evident. The chain is unkeyed, so a party with write access could rewrite everything from an edit onward. What nobody can do is change one record quietly.

The defence against a full rewrite is procedural and simple. The final hash goes to the client at close, and the client stores it somewhere we cannot reach.

How a record backs a proven finding

A finding in Arxiis is a claim plus an evidence pointer. The pointer names a ledger record. A deterministic function, not the model, then checks three things. The record exists. Its observed response contains the evidence the finding cites. Its hash still recomputes.

If all three hold, the finding is proven. If any fails, it is needs review. There is no third label, and the model does not choose. Our post on AI pentest false positives shows the function.

For bugs where the response alone shows nothing, the confirmation step is itself a ledger record, so the binding rule can point at it. When confirmation is not possible in the target environment, the finding stays needs review.

The report is then scrubbed against the labels. Impact language attached to a needs review finding is removed. Claims that go beyond what a record shows are cut, even on proven findings. The report states what was observed and nothing more.

This is the mechanism behind our measured 0% false-positive rate on XBOW-104 (19 October 2026). It is also why our cross-level solve rate on the same benchmark is 56% raw and 44% confirmed. The gap is mostly findings that were likely right but could not be bound to a record.

What a needs review finding is worth to an auditor

A needs review finding is not discarded, and it is not a lower severity. It is an observation that did not meet the evidence bar. Present it to an auditor as exactly that. List it separately, with whatever records exist, and add the outcome of human review once that review is done.

Mixing the two lists is the quickest way for a report to lose an auditor's trust. Keep them apart in the report, in the ticket system and in any summary for the board. Our board pentest report template shows one way to do that on a single page.

What auditors ask for, and where the answer lives

Auditor questions about a penetration test usually fall into six groups. Here is where each answer lives in an Arxiis engagement.

None of these answers depends on trusting our account of the test. Each one points to a document or a record the auditor can read for themselves.

Retention: how long, and what to keep

Agree the retention period in the authorization file before testing starts. Our rules of engagement template has a field for it. Set it with your legal and compliance teams, not with the tester. The right period depends on the regimes you report under and on what the evidence contains.

Evidence can contain data from the target environment. Minimise it at the source. Our template limits proof to a single synthetic record, which keeps sensitive data out of the evidence in the first place. Data that is never collected never needs protecting.

When the period ends, delete the evidence and keep a record that you did. The final hash and the deletion record are small. Together they let you show later that the evidence existed, was intact at hand-over, and was destroyed on schedule.

Hand-over at close

At the end of an engagement, the client receives the evidence, not just the report. Per-engagement evidence isolation means the hand-over is the whole of one engagement and nothing from any other. A short checklist covers it.

1. The signed authorization file, in the version in force at the end of the test, plus every amendment.
2. The report, with proven and needs review findings listed separately.
3. The execution ledger for the engagement.
4. The final ledger hash, recorded by the client in a system the client controls.
5. The agreed retention period and the named owner of deletion.

When findings are retested after a fix, keep the retest evidence alongside the original. The before and after should be readable side by side, under the same finding.

What this evidence does not prove

A ledger record proves that a specific request produced a specific response at a specific time, inside the authorized scope. It does not prove business impact, what an attacker would do next, or that data left the environment. We do not claim any of those, and the report scrubber removes language that implies them.

It also says nothing about the bugs the crew never found. On hard chained and blind attacks in XBOW-104, the crew solves between 0% and 42%, a known weak spot. A clean evidence trail for what was found is not coverage of what was not. An auditor should read the needs review list and any early-stop statement with that in mind.

Frequently asked questions

What counts as penetration testing evidence?

Any information that supports the tester's conclusions. That includes raw tool output, the requests and responses behind a finding, and records of who approved what. Evidence is strongest when you can show which run produced it and that it has not changed since. In Arxiis that means a record in the tamper-evident execution ledger.

What is proof of exploit in an Arxiis report?

A finding labelled proven. Its evidence pointer must resolve to a ledger record that exists, contains the evidence the finding cites, and still has a valid hash. A finding that fails any of those checks is labelled needs review.

Is a screenshot enough evidence for a pentest finding?

On its own it is weak. An engineer cannot replay it, and an auditor cannot tell when it was taken or whether it was changed. Keep the request, the response and a timestamped record that shows it has not been edited.

How long should penetration testing evidence be retained?

Agree it with your legal and compliance teams and write it into the authorization file before testing. Base it on the regimes you report under and the data the evidence contains. Delete it at the end of the period and keep a record of the deletion.

Can the client verify the Arxiis ledger after the engagement?

Yes. The ledger is tamper-evident: the client can recompute the hash chain and compare the final hash with the copy it stored at close. The chain is unkeyed, so storing that final hash independently is what protects against a full rewrite.