Learning long‑horizon, goal‑conditioned tasks requires an agent to reason over extended timescales and act across a broad state space. Hierarchical Reinforcement Learning (HRL) tackles both challenges by combining temporal (action) abstraction with spatial (state) abstraction. Representing temporally extended behavior as options reduces the effective decision horizon, while allowing different state abstractions at each level enables coarser data aggregation. However, these benefits hinge on learning a suitable action abstraction. Existing HRL methods either discard distinctions between options that are essential for optimal control—breaking the hierarchy—or retain unnecessary distinctions, preserving horizon reduction but forfeiting coarse state abstraction.
We formalize three desiderata for an action abstraction and introduce Q‑Shaped Options (QSO) to satisfy all of them. QSO builds on a hierarchy where each level has its own state‑value function, Q‑function, and policy. The action abstraction between consecutive levels is learned as a shared encoder shaped by the respective Q‑functions. The low‑level Q‑function treats the option as a goal, encouraging the encoder to keep distinctions needed for optimal control. The high‑level Q‑function treats the option as an action, pushing the encoder to discard superfluous distinctions.
Across offline goal‑conditioned locomotion and manipulation benchmarks, QSO learns semantically meaningful option spaces, outperforms baselines, and achieves non‑zero performance on tasks where all other evaluated algorithms fail.
Review: By jointly constraining the option encoder with low‑ and high‑level Q‑functions, QSO achieves a balanced abstraction that preserves critical control information while enabling higher‑level state abstraction, offering a robust advance for hierarchical RL.