Social preferences can foster cooperation in multi‑agent reinforcement learning, yet most existing approaches require agents to directly observe peers' reward signals. In real‑world interactions, agents typically only see others' actions and outcomes, without access to private rewards. We therefore introduce self‑referenced social preferences: each agent first learns a model of its own reward function, then applies this model to the observed state‑action‑transition tuples of other agents, obtaining an assessment of their outcomes from its own perspective. These self‑referenced assessments are fed into standard social‑preference mechanisms. We investigate two integration strategies: (1) augmenting the learning reward with the assessments; (2) weighting policy updates by the assessments. Experiments are conducted on three sequential social dilemmas—Escape Room, Clean Up, and Commons Harvest—requiring volunteering, public‑good contribution, and resource restraint respectively. Across all environments, agents learn to cooperate without observing true rewards, even in settings where independent learners fail to cooperate, and they often achieve more equitable splits of joint returns than agents with access to actual rewards. The optimal integration point depends on the preference type: inequity aversion works best when the assessment modifies the reward together with a value look‑ahead, whereas a purely benevolent preference benefits from weighting policy updates. Under partial observability, the policy‑weighting approach continues to support cooperation. These findings demonstrate that explicit access to other agents' reward signals is unnecessary for learning cooperative behavior; social preferences can instead be grounded in self‑referenced assessments derived from observed behavior.
Review