A preprint reports that SAPO, a training method for language-model agents, matched or slightly exceeded the study's PPO baseline while using fewer computing resources. Its abstract gives the headline comparison: 23.1% less peak GPU memory, meaning memory used by the graphics processor at its busiest point, and 24.8% less time per training iteration, with task success matched or slightly higher.
SAPO places the policy, state-value and action-value representations in one causal language model and uses natural autoregressive boundaries for value learning. In plain language, the model both chooses actions and estimates how promising a situation or a chosen action is.
A rollout is one sampled run through a task. SAPO derives turn-specific advantages and value targets from that single trajectory, then jointly optimizes clipped policy, state-value and on-policy SARSA action-value objectives in one shared update. The advantages are training signals that help compare what happened after a decision with what was expected.
The paper also tests how the scale of value targets affects training. It describes large logit differences, or gaps in the model's raw output scores, as a possible source of disruption to pretrained logits and interference with policy learning. It reports better performance when outcome rewards were scaled to 10.
Two benchmarks, three model scales
Experiments used Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct and Qwen3-14B-Instruct. The comparisons with PPO and GRPO used matched settings: a learning rate of 1 × 10−6, a KL coefficient of 0.01, 150 optimization steps, and episode caps of 50 ALFWorld turns and 15 WebShop turns.
The evaluation covered two benchmark environments: ALFWorld, with 3,827 task instances across six categories, and WebShop, with nearly 1.1 million products and 12,000 user instructions. Reinforcement-learning results were summarized as means and standard deviations over three random seeds.
Performance and computing costs
On ALFWorld, SAPO's aggregate success rate was 94.6% at 1.5B, 96.1% at 7B and 98.7% at 14B. It slightly exceeded PPO and was 14.9 to 21.8 percentage points above GRPO across the reported model scales.
On WebShop, SAPO's success rate stayed within 1.3 points of PPO and stood 7.2 to 17.5 points above GRPO.
The detailed results give peak-GPU-memory reductions ranging from 15.2% to 35.6% relative to PPO across the evaluated environments and model scales. Measured per-iteration runtime was 22.6% lower at 1.5B and 23.6% lower at 7B.
What changed when parts of the method were removed
The ablation tests examined what happened when parts of the update were removed. At 1.5B, the action-value-loss ablation produced 92.2% success rather than SAPO's 94.6%; at 7B, the corresponding figures were 91.2% and 96.1%. Without advantage normalization, the 1.5B result was 89.1%, and its standard deviation rose from 1.4 to 3.4; the 7B result was 95.5%.
Value estimates also tracked the sampled outcomes more closely as training progressed. Their Pearson correlations, a measure of how closely two quantities move together, with single-trajectory Monte Carlo returns, the eventual totals from one sampled run, rose from near zero to approximately 0.8, with most of the gain arriving within the first 50 training steps. These correlations are a diagnostic of the model's estimates in the tested runs, not a direct task-success score.
A narrower conclusion
The authors' conclusion is deliberately bounded: within the evaluated benchmarks, model scales and training settings, they describe SAPO as a low-cost PPO paradigm with no observable performance loss and say it may make PPO-style post-training more practical. That is a claim about the reported settings, not a finding that SAPO is superior to PPO across all tasks or model families.
The reproducibility statement says the appendix supplies ALFWorld and WebShop prompt templates, along with implementation and evaluation details intended to support reproduction. The evidence remains limited to those computational benchmarks, the three Qwen configurations and the training setup used here.
Paper data and sources
Original title: SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning
Authors: Dayang Liang, Lang Feng, Bo An, Yunlong Liu
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text