The Evidence Was Recovered. Did It Reach the Answer?

“... current evaluation frameworks mostly focus on the final success rate, revealing few insights during the process and failing to provide a deep understanding of the model abilities.” [1]

The execution was repaired.

The missing evidence was in the record.

The answer still had to use it.

In The Trace Was There. The Retry Was Blind. we gave a failed agent something more useful than “try again.” We gave it the failure recorded in its own execution. Trace-informed recovery met the published trace-support endpoint in 32 of 60 failed traces (53.3%); blind retry met it in 18 (30.0%). That answered the question we set: a recorded execution selects a better repair than an unconditioned resample. It left a different one open, and we closed the post on it. Recovering the evidence inside the execution is not the same thing as carrying that evidence into the answer a person reads.

This post is about the gap between those two. It is the third experiment in the series. The first, The Agent Was Right. The Evidence Wasn’t., asked whether an answer is supported by the execution that produced it. The second, linked above, asked whether feedback can repair an execution so that the missing evidence is recovered. This one asks what happens after the repair: does the new answer actually use it?

We define four terms. The execution is what actually happened: the tool calls, their results, the errors and the retries. The trace is the structured record of that execution. A ledger is an authoritative view of that record which can be checked against the answer. The answer is the prose the user sees.

This post is about whether evidence can exist in the repaired execution and still fail to reach the answer.

The recovery experiment left a second signal in the data.

The recovery experiment was designed to measure whether feedback could repair the execution. It was not designed to study what happened after that repair. But once we looked at the repaired runs at value level, another pattern appeared: among the 69 claims supported by the 24 value-grounded trace-informed executions, 33 were absent from the corresponding answers under the published scorer, the token-overlap rule this experiment replaces. Nearly half of the recovered evidence appeared not to make it through to the user.

That observation opens a different problem. Evidence recovery and answer synthesis happen at different points in the agent loop. A run can successfully acquire the missing evidence and still end with an answer that does not reflect the repaired state. So before building a larger experiment, we took those 24 executions apart in a diagnostic pass and asked where the apparent loss occurred.

Part of the gap was not an answer-synthesis failure at all.

In 9 of the 24 repaired runs, no model was ever asked to write a new answer after recovery.

Three stopped through a deferred path. Six were handled by a deterministic engine instead of a model, a fixed rule-based code path that builds the result from values already in the execution: four produced a result that way, and two produced nothing because the values they needed were absent. The recovery loop had a three-turn budget and no turn reserved for synthesis, so a run that spent its turns gathering evidence reached the end with nothing written.

Figure 1. What produced the answer in each of the 24 repaired runs. In the nine where no model was asked, the pipeline served the answer from before the repair.

That matters because the original artifacts combine two different phenomena: evidence that was available but not selected by the model for the answer, and evidence that never passed through a dedicated synthesis stage in the first place. That experiment did not need to distinguish those cases because its endpoint was recovery. This one does.

So we separated the stages. Recovery would answer one question: did the execution acquire the missing evidence? Synthesis would answer the next: given the repaired execution, what actually reaches the user?

Recovered evidence needs a guaranteed path through synthesis.

To analyse that we had to change the benchmark pipeline before running anything. Recovery and synthesis are now two stages. The agent can spend its recovery budget calling tools and gathering evidence, but once recovery ends the system reserves one final model turn to write the answer. A recovery that produces no answer is an explicit error rather than a silent fall back to an earlier response.

That is the architectural change, and it is the part that turned out to matter. The narrower question is what the stage needs. Once synthesis is guaranteed, does the model have to be handed the execution record explicitly, or is the ordinary conversation already enough?

To test it, we froze each repaired execution and generated two answers from the same run. No tool was called again and no recovery was repeated. Both answers used the same model, the same frozen execution and the same instruction, which named no claim or value and asked only for prose. The only difference was what the model saw at synthesis. In the first condition, ordinary synthesis, the model sees what an agent ordinarily holds at the end of a repaired run: the task, then the recorded execution as the conversation it was, every call in order with what came back, failures included.

In the second, ledger-conditioned synthesis, it sees exactly that plus one more message, a structured record of the same execution marking which calls succeeded, which failed, what arguments were used, and the result each successful call returned.

The ledger is not a cleaned-up version of the run. Failed calls stay in it, carrying their status and no result. Across the primary cohort there are 258 failed calls against 223 successful ones, so more than half of what the model is looking at is material it should know about and must not treat as evidence. We tested that on 24 repaired executions carrying 69 value-grounded claims. Each pair used the same model and the same frozen execution.

Ordinary synthesis carried 51 of the 69 supported values into the answer, 73.9%. With the explicit ledger it carried 56, 81.2%. At the trace level the paired gain was +5.9 percentage points.

The result is encouraging: the ledger condition carried more supported evidence into the answer. The sample is too small to call the effect established, but it is a clear signal worth testing at larger scale.

Table 1. Value-level propagation into the user-facing answer, 24 trace-informed repairs over the same frozen executions. Source: SourceryKit/MFS synthesis experiment.

The Trace Was There. The Retry Was Blind reported 32 repairs under its broader recovery measure. Here we use the stricter subset of 24 traces where the recovered value itself is present in the execution, so its propagation into the answer can be tested directly. The raw claim-count difference is 7.2 points; the reported +5.9-point effect is the paired per-trace estimate.

We repeated the same comparison on the 35 value-grounded repairs produced by the two control treatments, blind retry and failed-claim feedback.

Table 2. The same comparison on 35 repairs from the two control arms. Source: SourceryKit/MFS synthesis experiment.

Ordinary synthesis carried 80 of 100 values into the answer. With the ledger it carried 85. The paired gain was +4.8 percentage points, with a 95% interval from −2.3 to +14.2. The control repairs show the same encouraging pattern: the ledger condition carried more supported evidence into the answer. Note that this is not an independent replication. Those 35 episodes sit on 18 underlying traces, every one of which also appears in the primary cohort, so it is the same observation seen a second way.

Both cohorts show the same encouraging pattern: the ledger condition carried about five percentage points more supported evidence into the answer. The next step is to test whether that gain holds on a larger sample.

A separate no-answer-key diagnostic packet produced the same 56 of 69 on this cohort, which corroborates that figure against a different presentation of the same evidence without being an independent replication of it.

The result that matters for an operator is not the ledger. It is that the record says which kind of failure you are looking at.

An incomplete answer is not one failure. It is three, and they want different fixes.

Table 3. Three failures behind the same symptom. The interventions are what each failure implies, not something this experiment measured.

This experiment exposed the last two directly. In 9 of the 24 repaired runs, the evidence had been recovered but synthesis never happened. Once synthesis was guaranteed, a different problem remained: some supported evidence was still left out of the new answer.

From the final answer alone, those failures can look identical. The execution record tells them apart. It shows whether the system needs to recover missing evidence, run a synthesis step that never happened, or improve a synthesis that omitted evidence already available. That is why “just retry” is a poor default. The useful question is not only whether the answer is incomplete, but where the information was lost.

The main lesson is architectural: recovery and synthesis are different stages. Recovering the evidence does not complete the repair; the system must still turn that repaired state into a new answer.

The explicit ledger helps with that last step. Both cohorts showed the same encouraging direction even with a small experimental sample. Certainly, what is already clear is the diagnostic value of the record.

SourceryKit provides the evidence layer for that diagnosis by preserving the requests, responses, failures and ordering of the execution. It does not make synthesis correct; it makes the relationship between execution and answer checkable.

The first experiment in this series asked whether an answer was supported by its execution. The second asked whether that execution could guide recovery. This one closes the loop: a repair is not complete until the recovered evidence has a guaranteed path into a new answer.

Detect the gap. Repair the evidence. Carry the repair into the answer.

References

[1] C. Ma, J. Zhang, Z. Zhu, C. Yang, Y. Yang, Y. Jin, Z. Lan, L. Kong and J. He, “AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents,” arXiv:2401.13178, 2024. https://arxiv.org/abs/2401.13178