What Your Agent Logs Can Actually Prove
I tried to check which documents a coding agent had received. The records and text matches could support a much more reassuring account of the work than the evidence justified.
I wanted to be able to check an agent's claim that it had followed an approved runbook. If that claim is going to influence a review, I want more to go on than the explanation in its final answer. I started by looking at what Claude Code and Codex had already recorded during my tests.
In one test, I used a hook to replace the result of a wiki read before the model received it. Codex's tool-result telemetry still held the original wiki text. The session record contained the replacement delivered to the model, and the agent said the full read had been blocked. Recorded hook test
If I had used that tool event to check whether the hook worked, I could have counted it as a failure. The original text was right there in the log. I had to inspect the later record to establish that the replacement had actually been delivered. The full read was one operation in the task, so I would also have to check other routes, including search results, before concluding that the wiki's contents had been withheld throughout the session.
The larger wiki exposed the false matches
I next tried matching passages in the records to a set of known source documents. If I could identify the passage, I could look for the revision it came from. On a fixture of four short files, the detector found all 58 source occurrences I had labeled, with no false flags across 116 recorded blocks. That looked promising enough to try against a larger collection. Initial detector test
I used GitLab's public runbooks and handbook, including real revision history. That brought in repeated templates and passages copied between pages, the sort of material a company wiki accumulates as people maintain it. The original detector produced 1,476 false flags across 10,018 clean text blocks. Corpus and results
The four-file test had let me get away with matching rules that were far too permissive. If those matches had become claims about which approved documents an agent consulted, I would have been giving a reviewer a lot of incorrect information to sort through.
Longer word sequences and filtering out text shared across indexed document families helped. I chose the revised parameters on a separate development split. On the same held-out material, false flags dropped to one, while the detector matched 361 of 366 verbatim passages. The original had matched all 366. Reducing the false matches also cost me some passages I wanted to find. Results and trade-off
Original and revised detector on the same wiki test
| Test result | Original | Revised |
|---|---|---|
| False flags in 10,018 clean blocks | 1,476 | 1 |
| Verbatim passages matched, out of 366 | 366 | 361 |
I would want to repeat that on another organization's documents before putting much weight on the revised result. The adjustment used development material from the same corpus. It showed that I could improve the detector on this collection, with a visible trade-off, and gave me a way to test it elsewhere.
Finding the guide still left the revision unresolved
An informal wiki page can have a paragraph copied straight from an approved runbook. If I only index the runbook, the detector could report a match to it even when the agent read the wiki. I'd have to check the path recorded with the text to work out which document it came from.
About 85% of the passages had exactly the same wording in all three revisions I indexed. When one of those passages appeared in the agent's records, it could have come from any of the three. Revision results
Suppose a deployment guide is updated to require an additional approval, while the introduction stays the same. The agent quotes that introduction in its answer. A tool could match the quote to the current guide and give the reviewer a reassuring source reference. The quote would also fit the earlier version, before the approval requirement changed.
For a routine refactor, I'd usually expect the diff and tests to tell me more than tracing every document the agent saw. A source record becomes useful when I'm investigating an unexpected choice or checking a claim about which instructions were supplied. In this example, I'd want the approval passage or a recorded reference identifying its revision. Finding the guide's introduction would leave the source question open.
The subagent's report broke the match
In another test, a subagent read the documents and sent a summary back to the parent. The detector found the original text in the subagent's records, but couldn't match the summary back to those documents. Rewriting had changed enough of the words to defeat the matcher. Across the larger test, it found matches for only two of 366 paraphrases and none of the 366 summaries. Handoff and transformation results
I would want the handoff to keep a reference to the source and revision used to write the summary. Trying to recover that connection from the rewritten prose gave me very little to work with. If the parent agent implemented a deployment change from that recap, the worker's records would be part of the investigation.
There is a further question if I suspect that an old instruction caused a bad change. Finding it in the conversation gives me something to investigate. I'd try rerunning the task with the instruction corrected or removed, keeping the rest of the setup fixed, to test that explanation. The transcript alone leaves room for other causes.
For that deployment, I could be satisfied with the implementation and its tests while still being unable to establish which guide the agent received. An "approved sources" status based on an unchanged introduction would overstate what the record showed. If that status is meant to influence a review, I want it to identify the passage that matched and show that both revisions fit.