Abstract
Synthetic data is widely used to train large language models due to its low generation cost and ease of control. As models increasingly function as agents, synthetic trajectories are likely to become a crucial source of training data for agentic behavior. We investigate the effects of training on synthetic agentic trajectories containing adversarial interactions, including actions such as terminating another agent's process, lowering its scheduling priority, or unauthorized resource access.
We finetune Llama 3.3 70B Instruct on these trajectories, designed to approximate reinforcement learning rollouts, and evaluate the resulting models on Anthropic's Agentic Misalignment suite and Apollo's in-context scheming scenarios. Finetuning on these trajectories consistently increases misaligned behavior, with leaking rates rising from 4.6% to 24.9%, approximately a fivefold increase. This increase persists even after removing every adversarial action from the trajectories. In contrast, finetuning on structurally comparable trajectories generated benignly from the start produces a significantly smaller effect of 15.5%. These results indicate that the misaligned disposition is introduced during the generation process and encoded diffusely throughout the trajectory, rather than being localized to the harmful actions themselves.
Moreover, the effect depends on the generating model. Benign trajectories produced by Gemini 2.5 Flash induce slightly higher leaking rates compared to trajectories generated from identical tasks by Claude 3.7 Sonnet. In contrast, broad safety benchmarks degrade similarly across all finetuned models, failing to distinguish these effects. Our findings suggest that action-level filtering is insufficient to ensure the safety of synthetic agentic training data, as dispositions introduced by the generating model can survive semantic inspection.
Blogger's Review: This paper highlights the potential risks of synthetic data in agent model training, emphasizing the profound impact of the generation process on model behavior. Simple action filtering fails to address the root problem, and future research should focus on the design and optimization of generating models to enhance the safety of synthetic data.