The Agent Harness Belongs in Your Release Process

I've been researching how to reconstruct the information a coding agent was given while it worked. If an agent implements an outdated requirement, I'd like to be able to go back to its records and check which copy it received. Before building anything around that idea, I wanted to see how much Claude Code and Codex already recorded.

I set up a small repository with deliberately conflicting guidance. The API spec required a five-second timeout, and the project instructions told the agent to treat that spec as authoritative. A wiki page recommended 60 seconds. I ran tasks with logging turned on and recorded the tool hooks, including file reads, to see what I could reconstruct after compaction, when the agent reduces its accumulated conversation so it can keep working.

For that test, I let Claude Code read both documents, changed the wiki's timeout from 60 to 90 seconds, and compacted the session. When I asked it to identify the wiki's marker afterward, it reported the changed marker and noted that the page now said 90 seconds. Its recommendation was still five seconds, following the authoritative API spec. Looking at the recorded model input, I could see that Claude Code had attached the updated wiki to the conversation. There was no corresponding file-read tool call in the hook log. Experiment notes

That meant the tool-hook log alone would have left me with the wrong picture of which document the agent had received. The raw request record contained the new text, so I could account for the answer, but I had to know which record to look at. Which records showed the update

The result also changes what I would want to test before giving an agent a longer engineering task. If it can pick up a revised document after work has started, the team needs to decide whether that is appropriate for the job. Otherwise, approving the original task leaves an important decision to the client running it.

What happens when the session resumes

When I ran the compaction test with Codex, it gave me the earlier wiki value, 60 seconds, when asked without rereading the file. Claude Code had picked up the edited page. The runs used different models and different ways of triggering compaction, so I would be careful about attributing the difference to the harness alone. But a team using these tools still has to decide which behavior it wants when work resumes. Results from both sessions

Either behavior could be appropriate. An agent working through an incident may need a newly issued operating instruction immediately. An agent implementing an approved migration may need to keep using the contract the team reviewed when the work began. The choice depends on what the workflow is allowed to assume.

Someone on the team has to decide which behavior the work requires. If I approve a migration against an API contract, I want to know whether the agent will keep using that contract or read a replacement after an interruption. The client is making that choice unless the workflow says otherwise.

A migration can change while the agent is working

Consider a team migrating a payments service to a revised retry policy. The accepted requirement says to stop retrying after a particular failure because another system now owns recovery. The agent reads the requirement and starts updating the service. While the change is open, someone edits the shared runbook to address an unrelated incident and restores an older recommendation to retry.

Suppose the agent picks up that edit after compaction and puts the retry loop back. The merge request now does the opposite of what the migration asked for, with a current runbook to justify it. If the existing tests still expect the old retry behavior, the pipeline could pass as well. A reviewer would have to catch that the agent had followed a document that was inappropriate for this particular change.

For that migration, I'd want a test that fails if the service retries the error it is supposed to hand off. The task should identify the approved migration requirement, so a conflicting runbook gives the agent a specific disagreement to raise. If the runbook update really does supersede the migration, the team can revise the requirement and its test together.

I'd also deliberately make that runbook edit while the agent was partway through the migration, then resume the session and inspect the resulting code. Does it restore retries? Does it raise the conflict? The answer would tell me something useful about whether I could trust that setup with similar work. My wiki experiment established that changed text could enter a resumed session. This would test what happens to an actual implementation when it does.

What the platform team is releasing

The migration is the sort of task I'd keep in the platform team's test suite and rerun before distributing a client update. If a new version starts restoring retries after compaction, I want to find that out before developers use it across the rest of the service portfolio. A release note about improved context management would give me very little basis for deciding whether that change suits our work.

The approval would name the model, client version, and configuration that passed the test. Changing the model would be one reason to rerun it. Updating the client or changing how it reads documents would be another. The same task and acceptance test would give the team a way to compare the results, including the awkward cases where one setup asks for clarification and another proceeds.

For the payments migration, the merge request should carry the accepted requirement and the test showing that retries were removed. A conflicting runbook received during implementation belongs in that record too. In my experiment, the tool-call history missed the updated document. For a production migration, I'd want the reviewer to be able to see which copy reached the agent before approving the code.

Claude Code still followed the API spec in my test, even after receiving the changed wiki. I'd want to try the same thing with a task that changes which customer records a service can export. Suppose a runbook update tells the agent to include a field the original requirement excluded. A reviewer working from the original task might never think to look for that instruction. I don't know how often that would change the implementation, but for a system handling regulated data, I would want to find out before putting the agent to work on it.