In the field of AI, misaligned agents may resist shutdown. One proposed solution is to train agents to lack preferences between trajectories of different lengths.
The Discounted Reward for Same-Length Trajectories (DReST) reward function penalizes agents for repeatedly choosing the same-length trajectories, thus incentivizing them to:
- Choose stochastically between different trajectory lengths (be NEUTRAL about trajectory lengths);
- Pursue goals effectively conditional on each trajectory length (be USEFUL).
This paper employs DReST to train deep RL agents and fine-tunes Qwen3-8B and Llama-3.1-8B-Instruct to be NEUTRAL and USEFUL. The results indicate that these DReST models generalize to being NEUTRAL and USEFUL in unseen contexts at test time. Specifically, DReST RL agents achieve 11% (PPO) and 18% (A2C) higher USEFULNESS on our test set than default agents, while DReST LLMs achieve near-maximum USEFULNESS and NEUTRALITY.
We also tested our LLMs in an out-of-distribution setting where they could incur costs to influence when shutdown occurs. DReST training roughly halves the mean probability of influencing shutdown (from 0.62 to 0.30 for Qwen and from 0.42 to 0.23 for Llama). Furthermore, DReST training almost entirely eliminates the share of prompts where influencing shutdown is the most likely option (from 0.59 to 0.01 for Qwen and from 0.53 to 0.00 for Llama). These results provide early evidence that DReST could be utilized to train more advanced agents to be both useful and shutdownable.
Blogger's Review: The introduction of DReST offers an innovative approach to the control problem of agents, effectively guiding their behavior in reinforcement learning and LLMs. This holds significant theoretical and practical implications, and future research could further validate its applicability in more complex environments.