Auditing AI Agent Workflows with Execution Evidence

The diff is one piece of evidence

A Git diff shows persistent source changes. It cannot show that an install script tried to contact an unknown host, a test accessed a local service, or a browser opened an authenticated page. An agent-written summary may describe those events, but it is testimony from the subject of review. The useful record pairs the patch with independently captured execution facts.

Consider a small agent change in this repository. The host creates a box pinned to the current revision; the agent works in its shell, and the test runs under the same box policy:

h5i box create review-demo --from HEAD --profile agent-claude
h5i box status review-demo
h5i box shell review-demo
# Inside the box: ask the agent to make the change, then exit.
h5i box run review-demo -- cargo test

This sequence gives the later review a base revision, an enforced policy, and an observed test exit. It does not say that the patch is correct or that the test covers it. The workflow guide covers profile setup and the full procedure.

Ask each record the question it can answer

Command records show invocations, exits, and relevant output; browser records show navigation and requests; policy decisions show allowed and denied connections; a patch shows proposed source changes. A receipt can connect these records to one named run and report the isolation tier. None of these alone proves that the code is correct or that the agent understood the task.

Before reading the agent's explanation, inspect the run from the host:

h5i box status review-demo
h5i box log review-demo
h5i box diff review-demo

status reports the boundary that was actually enforced, log shows recorded events, and diff shows proposed source changes. During a live run, h5i box watch review-demo --deny-only can stream refusals until stopped. If the agent said “all tests passed,” compare that claim with the recorded cargo test exit before reading the patch.

The distinction between observation and interpretation is especially important for web testing. A browser record can show the request that reached an invoice endpoint and the response it received. It cannot, without an application policy, establish that the response was unauthorized. Likewise, a command exit code can show that a test process returned zero; it cannot establish that the test covered the changed behavior. The report should point back to each source record instead of collapsing them into one confident sentence.

Absence also matters. If browser recording was unavailable, the report should say so. Otherwise a blank browser log can be mistaken for a session with no browser activity. A denied request deserves a place in the review, even when it never changed a file.

A useful receipt therefore includes not just successful events but the conditions under which events were captured: the chosen isolation tier, network-policy status, evidence lanes, and known gaps. If a browser crashed, quietly starting a fresh session would erase a meaningful boundary in the story. Recording the end of that session makes the gap visible to the reviewer.

Keep observation and acceptance outside

If the agent can edit its audit trail, the trail is merely another output file. Store records outside the box's writable paths and resolve its policy outside them too. The box can propose a patch; a host-side export gate packages the patch, report, and receipt for human review. Applying that patch to the parent repository is a separate decision.

If the task used a browser, keep it and the local dev server in the same box so their requests belong to the same boundary. h5i ui lets a reviewer inspect active sessions and records; its display is a view of captured evidence, not an additional guarantee.

Export is more than file copying. It is the point where proposed work leaves a lower-trust workspace and becomes something a person may apply elsewhere. The record should identify the base revision so the reviewer can recognize drift, and the patch should remain separate from the evidence explaining how it was produced. A detached pull-request workspace cannot safely apply its own changes to a parent checkout by assumption; the reviewer must choose the destination and reconcile conflicts.

h5i box export review-demo --out ./review-demo
git apply --check ./review-demo/patch.diff

Read review-demo/report.md beside review-demo/receipt.json and review-demo/patch.diff. The check asks whether Git can apply the patch; it does not apply it or approve the change. A failed check may simply mean the destination has moved since the pinned base.

A useful review order

1. Check the enforced isolation tier, policy, and any missing evidence.
2. Read denied network or file actions and failed commands.
3. Inspect browser actions, requests, and any human takeovers.
4. Compare executed tests and outcomes with the agent's claims.
5. Only then review the diff and decide what to import.

This order keeps the proposal from framing the evidence before the reviewer has seen the run. It also exposes a common mismatch: a plausible patch with no test execution or a successful test report that actually belongs to an earlier state.

The limits of a receipt

Host-held evidence prevents the boxed workload from rewriting its own account. It does not prove that the host owner did not alter the bundle later, and it does not attest to facts no sensor captured. The isolation tier and evidence lanes must therefore travel with the report. A claim such as "nothing suspicious happened" is too broad; "no denied egress was recorded while this network policy was enforced" is reviewable.

There is also a practical cost. Capturing more detail can expose secrets in logs and create more material for a reviewer to inspect. A good system keeps the evidence needed to answer the task's security questions, restricts who can read it, and reports where collection was incomplete. The point is an honest account with a bounded claim, not an ever-growing transcript.

Execution evidence makes agent work easier to inspect and reject. It does not turn a human approval into a formality. The reviewer still decides whether the code, the observed behavior, and the evidence gaps are acceptable.