We investigate how to improve decision policies using observational data so that more effective complex structured actions—such as text messages or images—can be generated. Two major obstacles arise: (i) action‑level overlap is often too weak, making off‑policy evaluation and improvement unreliable; (ii) generating new structured actions typically requires costly training or fine‑tuning of models (e.g., large language models) and can be ill‑defined because of information loss in action representations. To tackle both issues, we introduce PolicyLift, a three‑stage framework for generative policy improvement with structured actions from observational data. In Stage 1, PolicyLift learns a compact set of discrete action codes that capture high‑level strategies in the observed action space and are specifically designed to support policy improvement under limited overlap. In Stage 2, it learns a context‑dependent policy over these codes. At deployment, Stage 3 “lifts” the learned code‑level policy back to the original action space by generating actions from the historical behavior system conditioned on the selected code. We provide a theoretical guarantee that improving the policy over action codes necessarily improves the induced policy in the original action space. Empirical results on synthetic benchmarks and real‑world text generation tasks show consistent policy gains over the behavior policy.
Review: PolicyLift’s abstraction of the action space and offline learning at the code level elegantly sidestep both overlap scarcity and high‑cost generation, offering an interpretable and efficient route for structured decision making.