Pluralistic alignment demands that AI systems learn from diverse populations whose preferences may be heterogeneous or even conflicting, and then personalize behavior for each user. Existing approaches usually assume access to annotator identities and rely on powerful pretrained representations to capture individual preference variations. We introduce Preference‑PCA Pluralistic Learning (P3L), a framework that jointly learns trajectory embeddings and a shared basis of preference directions via PCA decomposition of heterogeneous crowd feedback, without requiring annotator identities.
At test time, P3L enables rapid and accurate personalized reward learning and downstream policy optimization. Experiments on several embodied control tasks demonstrate that P3L provides steerable personal alignment that generalizes beyond the offline dataset to out‑of‑distribution new users, achieves competitive RL and test performance, substantially reduces computational cost, and supports variable adaptation query budgets.
Review