Experiment Notes on Tracing Documents Through Agent Records
Setup and results supporting What Your Agent Logs Can Actually Prove.
These notes describe the recorded tests behind the article. The work compared native agent records and tested a word-based document detector on a small fixture and a larger public wiki corpus. It also examined what could be attributed to a particular document revision.
The hook replacement test
In the native-capture experiment, a hook replaced the tool output from a wiki read before that output reached the model. Codex's tool-result telemetry contained the original text. Its session rollout held the original tool event and the replacement in the model-facing record. The agent's answer reported that the full wiki read had been blocked.
The observation concerns that read operation. Other operations in the task included a search of the documents, so blocking the full read does not establish that every word from the wiki was withheld throughout the session.
The original tool event supports a statement about what the tool returned. The model-facing record supports a statement about what the model received at that point in the task. The test used Codex 0.166.0-alpha.6.
File access and recorded text
The later live fixture contained twelve tracked runbooks. In the main Claude Code and Codex sessions, a trace of the agent's process tree showed all twelve files opened. Identifiable text from seven appeared in the records inspected for each session.
The evaluation used distinctive raw-text spans to identify document occurrences. That method provides a bounded count of identifiable text. An absence of matching words cannot establish that no information from a document arrived through a paraphrase or another transformation.
The initial fixture
The pilot used four short instruction and documentation files, including revisions created during the native-capture tests. It matched normalized lines and sequences of six words. Distinctive test-marker lines were excluded from matching.
Across 116 recorded blocks, it identified all 58 labeled source occurrences, with no false flags. The source documents were known in advance, and the fixture offered a small range of repeated or unrelated text against which to test false matches.
The larger wiki test
The larger corpus used GitLab's public runbooks and engineering and security handbook pages. It enrolled 245 runbook document families with their last three real Git revisions. Additional runbooks and handbook pages supplied material outside the enrolled source set.
Document families and clean documents were separated into development and held-out test splits. Parameters for the revised detector were selected on the development split before scoring the test material. Both the original and revised detectors were evaluated on the same held-out examples.
The test split included 183 enrolled families, 366 source passages, and 10,018 clean blocks. Clean blocks were non-overlapping windows of 1,500 characters. The source passages were also tested after transformations such as reformatting, excerpting, paraphrasing, and summarization.
Detector results
The original detector used the pilot's matching method without recalibration. The revised detector used ten-word sequences, discarded sequences shared across enrolled families, and required six distinct matches. A decoding stage handled some formatting and encoding transformations.
| Result on held-out material | Original | Revised with decoding |
|---|---|---|
| False flags in 10,018 clean blocks | 1,476 | 1 |
| Verbatim passages matched, out of 366 | 366 | 361 |
The revised detector also flagged 127 clean blocks containing genuine shared text, including copied passages and templates. These were adjudicated separately from the single false flag. A shared passage can support several possible source documents, which is why a correct text match can still leave source attribution unresolved.
The reduction in false flags came with missed verbatim passages. Repeated template sections could lose the shared word sequences on which matching depended. These results come from one organization's public runbooks and handbook, with parameters selected using development material from that corpus.
Revision attribution
About 85% of sampled passages were identical across the three indexed revisions. Their text therefore identified a set of compatible revisions. The revised detector returned the exact compatible revision set for 99.4% of its verbatim detections.
These results concern the revisions compatible with the recorded wording. Selecting one revision from a shared passage requires additional evidence, such as a versioned source reference or text unique to that revision.
Summaries and paraphrases
On the held-out replay, the revised detector matched two of 366 model-written paraphrases and none of 366 summaries. The model-written versions were generated from the source passages. Their origin was known from the test construction.
In a separate live Claude Code session, a subagent read source documents and returned a paraphrased report. Matching identified source text in the worker's records and missed the report delivered to the parent. The live evaluation covered three sessions in total, using Claude Code 2.1.318 and Codex 0.166.0-alpha.6 across the main and delegated runs.
The tests establish limitations of this detector and the inspected records. An observed source passage establishes that text was present at a recorded stage. Investigating whether it caused a particular implementation choice would require a further test of that explanation.