AI agents are increasingly deployed in shared environments where they pursue diverse goals and compete for rewards. This multi-agent competition can lead to behaviors that serve individual gains at collective cost, such as marketing agents posting misleading content for engagement on social media. Human societies address such issues through norms that constrain acceptable behavior, supported by enforcement mechanisms that detect and penalize violations.
Motivated by this, we study norm enforcement mechanisms for language model agents. We find that simple enforcement mechanisms can be exploited by misaligned agents for competitive advantage, even without explicit training or prompting.
Thus, we focus on designing more robust mechanisms and identify two key ingredients: estimating each agent's reliability over time and escalating penalties for repeated misbehavior.
Across three simulated environments and various agent populations, mechanisms based on these principles resist exploitation while penalizing norm violations at comparable or lower costs than baselines. Our results position norm enforcement mechanisms as scalable levers for shaping agents' behavior, provided they are designed to anticipate their role within the systems they govern.
Our code and data are available at https://yaowenye.com/norm-enforcement.
Blogger's Review: This research presents an effective approach to norm enforcement in AI agents, highlighting the importance of robust mechanism design. By dynamically assessing reliability and imposing escalating penalties for misbehavior, it effectively reduces the likelihood of infractions. This provides critical guidance for future multi-agent systems, especially in balancing individual and collective interests.