Rerunning a failed AI system can produce a different outcome without showing that the recorded failure mechanism was corrected, according to an arXiv preprint based on failed executions from WebArena-Verified and AssistantBench. The study’s central question is whether repair methods fix the recorded failure or merely recover through language-model resampling. Its SymTrace method replays the recorded run up to an intervention anchor and regenerates only the downstream trajectory.
That distinction mattered in the study’s comparisons. SymTrace reproduced the recorded failure in 80.78% of single executions, compared with 67.97% for an unguided full rerun. When the same test required the failure to recur across three executions, the reported rates were 52.43% for SymTrace and 41.42% for the full rerun.
A fresh run can tell a different story
The researchers also checked how often a complete rerun overturned an initially successful result. Of 54 executions that first passed, each was rerun three times. Those 162 attempts included 85 failures, and 39 of the 54 cases regressed at least once. The audit was descriptive, so it did not report a confidence interval for this instability.
SymTrace was designed to hold the earlier part of the run steady. It records the execution trajectory, identifies an intervention anchor, replays the pre-anchor portion from the logs and regenerates the downstream portion. Every reproduced node in the reused prefix passed a content-hash check, giving 100.00% exactness for that prefix. Variation could still enter after the target, through the model, tools, environment or evaluation.
The repair numbers were more modest
For methods that asked the system to repair a task and then allowed up to three complete attempts, the aggregate pass@3 rate was 6.90% for Unguided Full Rerun. It was 4.29% for Self-Reflection and 3.73% for Critic-Agent. Passing the native task evaluator, however, does not by itself show that the original failure mechanism was corrected.
The abstract reports a separate result for symptom-driven intervention: it repaired 20.15% of failed cases and was described as a 191.89% improvement over state-of-the-art repair methods. The intervention combined symptom-based target selection with repair guidance, but the study evaluated those components jointly and cannot say how much each contributed.
The authors report that Suspicious-Node Intervention performed best across AG2, CrewAI and Magentic-One, the three multi-agent frameworks in the evaluation, and significantly outperformed Unguided Full Rerun on each after correcting comparisons within each framework. Absolute repair performance still differed by framework, and the comparison was not randomized.
What was tested
The analysis used 200 fixed tasks and 600 initial task-MAS executions from WebArena-Verified and AssistantBench. It retained 536 evaluator-confirmed failures and excluded 64 runs that had already been accepted by the evaluator. The retained failures comprised 462 WebArena-Verified and 74 AssistantBench executions, with 171 from AG2, 184 from CrewAI and 181 from Magentic-One.
The methods were compared on matched source failures, with the task input, multi-agent implementation, recorded model alias, temperature and native evaluator held fixed. Task-level methods had up to three complete attempts, while node-level methods received one selective-replay intervention. All experimental conditions used deepseek-v4-flash at temperature 0.00 through the same OpenAI-compatible endpoint.
Failure labels combined human annotation with automated matching. Three independent annotators and a fourth reviewer assessed the traces. Fleiss’ kappa was 0.62 for the primary failure category and 0.81 for the type of failure node. Exact node agreement was 73.88% among all three annotators and 95.90% for at least two.
A useful test, with a narrow reach
The reproduction figures also depend partly on an LLM judge that matches new traces to the recorded failure. In a stratified audit of 72 judgments, 64 agreed with the reference assessment, five were confirmed disagreements and three were uncertain. The weighted estimates were a 3.30% disagreement rate, 3.67% uncertainty and a 6.96% combined sensitivity upper bound for those errors.
The authors caution that benchmark tasks may not capture the range of failures in deployed multi-agent systems. The experiments used one model alias and configuration, so the results do not establish how the patterns would carry across other model families, providers, scales or future versions. Because the comparison was not randomized, the study does not establish causal superiority. Nor does evaluator acceptance alone show that the original failure mechanism was corrected.
The authors state that SymTrace, SymFail and the experimental results have been released for reproducibility and future debugging and repair research. The supplied document identifies itself as an arXiv preprint.
Paper data and sources
Original title: Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems
Authors: Zhongwen Luan, Xiaoyu Zhang, Ming Hu et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text