Co-Evolving Acting and Forecasting for Language Agents

Abstract

Agentic reinforcement learning learns only from a terminal scalar reward, so most of a long rollout goes unused. Per-step value targets all derive from that one scalar, and next-observation targets are long, noisy, and reach only one step ahead. CoPE adds multi-step action targets: on successful trajectories, the policy backbone is also trained to forecast the actions that follow each history, so acting and forecasting co-evolve in one model. At inference it still acts one step at a time, with no overhead. Across ALFWorld, ScienceWorld, WebShop, τ²-bench, and AppWorld, CoPE outperforms GRPO and predictive-auxiliary baselines with up to 2.0× the sample efficiency, using no extra models, pretraining stages, or inference-time cost.

Why predict action sequences?

Method

Replay

Replay

Acting helps forecasting via trajectory collection

The forecasting targets come from the policy itself: they are the action chunks of the successful rollouts it collects. For a fixed rollout policy, the forecasting loss is minimized by the policy's own chunks reweighted by their chance of success, which the success filter implements by rejection sampling. That target is always a step ahead of the policy that produced it, and a better policy supplies more successful rollouts, so forecasting gets more and better targets as training goes on.

Forecasting helps acting via implicit temporal ensembling

Read by column in the grid above, each action of a successful rollout is forecast from the K preceding histories as well as its own, so it receives K + 1 supervision signals instead of the single one GRPO gives it. These forecasts agree with acting: they are conditionals of the same hindsight distribution, so the best forecast of a_t from an earlier history equals the GRPO target marginalized over the observations in between. Because forecaster and policy share parameters, those forecasts are folded into π_θ: an ensemble over time that costs nothing at inference.

Results

Cost

Inference: no extra cost. The forecasting objective is used only in training. At inference the agent acts with π_θ alone, one action per step and without the forecasting instruction, so a CoPE policy costs exactly what a GRPO-trained policy costs.

Training: +11% per step, −9% end to end. On ScienceWorld the forecasting pass adds 11% time per training step on average, while the shorter rollouts a better policy produces cut the measured end-to-end step time by 9% compared with GRPO. There is no separate model and no pretraining stage.

BibTeX

@article{yu2026cope,
  title   = {Co-Evolving Acting and Forecasting for Language Agents},
  author  = {Yu, Haofei and Chen, Yixing and Zhao, Yining and Ye, Xuyan and
             Liu, Yuxiang and Hua, Zhigang and Xie, Yan and Yang, Shuang and
             You, Jiaxuan},
  journal = {arXiv preprint arXiv:2610.20597},
  year    = {2026}
}