With the rapid progress of large language models (LLMs), agent systems powered by LLMs show great promise for tackling complex tasks that require multi‑step reasoning or tool interaction. To fully exploit LLMs within a well‑designed agent paradigm, post‑training across diverse agent scenarios is often needed to boost performance.\
Among various post‑training techniques, alignment methods such as PPO, DPO, DIL and GRPO have become popular because they can penalize negative samples while keeping training complexity manageable. However, most of these methods focus on simple single‑turn tasks, leaving room for improvement on complex multi‑turn problems.\
We introduce Turn‑level Multiscale Density Ratio Estimation (tlm‑DRE). The key idea is to assign distinct weights to each turn and perform asymmetric token‑level training based on the gap between positive and negative sample spaces across multiple turns. Specifically:\
- For each turn $t$, compute the density ratio of positive to negative samples $r_t = \frac{p_t(x)}{q_t(x)}$;\
- Dynamically adjust the loss weight $w_t$ according to $r_t$, giving stronger learning signals when the distribution gap is larger;\
- At the token level, apply standard cross‑entropy to positive samples while adding a $w_t$‑scaled penalty to negative samples, achieving asymmetric learning.\
We evaluate tlm‑DRE on a suite of public agent benchmarks, covering tool use, dialogue planning, code generation and more. The results show that tlm‑DRE matches or slightly outperforms traditional alignment methods overall, and it excels in multi‑turn reasoning scenarios, maintaining high success rates both in‑domain and out‑of‑domain.\
The advantages of the approach are:\
- Turn‑level weighting captures the varying importance of information across long sequences;\
- Asymmetric token‑level training amplifies the signal gap between positive and negative samples;\
- Training overhead is comparable to existing alignment methods, making it easy to plug into current LLM training pipelines.\
Review: tlm‑DRE provides a fine‑grained alignment mechanism for multi‑turn tasks, addressing the gap left by single‑turn alignment techniques. Its promising results merit further exploration across a broader range of LLM‑driven agent systems.