Traditional multimodal preference alignment follows a reward‑based RLHF pipeline: first a reward model is trained to capture human preferences, then it scores on‑policy responses and guides the response model’s generation. This reduces preference supervision to a scalar signal, so the model learns to produce answers with higher scores but rarely learns the underlying reasons for human preferences.
To examine whether endowing the response model with an internal capability to understand preferences can improve generation, we introduce PrefMind, an online dual‑agent reinforcement learning framework. It consists of two trainable agents: a Generator (response generator) and a Critic (preference evaluator). For each interaction, a pair of candidate responses is presented; the agents analyze and discuss which response better aligns with human preference. Human labels supervise their judgments and provide corrections. When both the Generator and the corrected Critic correctly identify the human‑preferred response, the Generator produces an improved answer, and the Critic compares this answer with the original preferred one to generate a generation reward. Both agents are jointly updated from interactions sampled under their current policies.
Through this iterative process, PrefMind trains the Generator to acquire an explicit understanding of human preferences via judgment and correction, and to translate that understanding into higher‑quality responses. Experiments across three model scales demonstrate: (1) a notable boost in standalone preference judgment accuracy; (2) downstream response generation quality improvements as well.
Review