TY - GEN
T1 - Offline Multi-Agent Preference-Based Reinforcement Learning with Agent-aware Direct Preference Optimization
AU - Kou, Qian
AU - Li, Mingyang
AU - Liu, Zeyang
AU - Qian, Long
AU - Chen, Zhuoran
AU - Wan, Lipeng
AU - Chen, Xingyu
AU - Lan, Xuguang
N1 - Publisher Copyright:
© 2025 International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org).
PY - 2025
Y1 - 2025
N2 - Multi-agent Preference-Based Reinforcement Learning (MAPbRL) is promising in offline policy learning by leveraging human preferences to replace complex manual reward designing. Current MAPbRL methods use complicated structures to realize better reward modeling with off-the-shelf MARL algorithms and obtain the joint policy based on it. However, it faces a severe preference-behavior mismatch problem stemming from the instability of RL training and global-local preference inconsistency datasets in offline MARL, resulting in potential suboptimal policy convergence. To address this problem, we propose Agent-aware Multi-Agent Direct Preference Optimization (AMADPO) by utilizing a multi-agent preference predictor to guide agent-aware direct optimization from imbalanced preference labels, which can learn coordination policy from both positive and negative segments. Experimental results in SMAC environment show substantial improvements in global-local preference inconsistency datasets, demonstrating the effectiveness of AMADPO in solving the preference-behavior mismatch problem.
AB - Multi-agent Preference-Based Reinforcement Learning (MAPbRL) is promising in offline policy learning by leveraging human preferences to replace complex manual reward designing. Current MAPbRL methods use complicated structures to realize better reward modeling with off-the-shelf MARL algorithms and obtain the joint policy based on it. However, it faces a severe preference-behavior mismatch problem stemming from the instability of RL training and global-local preference inconsistency datasets in offline MARL, resulting in potential suboptimal policy convergence. To address this problem, we propose Agent-aware Multi-Agent Direct Preference Optimization (AMADPO) by utilizing a multi-agent preference predictor to guide agent-aware direct optimization from imbalanced preference labels, which can learn coordination policy from both positive and negative segments. Experimental results in SMAC environment show substantial improvements in global-local preference inconsistency datasets, demonstrating the effectiveness of AMADPO in solving the preference-behavior mismatch problem.
KW - Multi-agent reinforcement learning
KW - Offline reinforcement learning
KW - Preference-based reinforcement learning
UR - https://www.scopus.com/pages/publications/105009790925
M3 - 会议稿件
AN - SCOPUS:105009790925
T3 - Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS
SP - 1181
EP - 1190
BT - Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2025
A2 - Vorobeychik, Yevgeniy
A2 - Das, Sanmay
A2 - Nowe, Ann
PB - International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS)
T2 - 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2025
Y2 - 19 May 2025 through 23 May 2025
ER -