Abstract
Improving the task performance of Large Language Models (LLMs) is essential, yet scaling these models faces significant challenges such as diminishing returns and high costs. Multi-Agent Systems (MAS) offer a promising solution by distributing tasks among specialized agents to improve overall task performance. This can reduce training costs at the expense of increased test time due to the discussion and decision-making process.
The decision protocol is a critical component of MAS because it specifies how multiple agents collaborate to create a final solution. This thesis introduces the Multi-Agent LLM (MALLM) framework, which implements and evaluates various decision protocols, namely voting, consensus, and judge decision mechanisms, to simulate multi-agent discussions for conversational task solving. Unlike previous work that used a single decision protocol or tested them on limited datasets, this study systematically examines their impact on a diverse set of tasks, ranging from knowledge-based datasets (MMLU, MMLU-Pro, GPQA) to logic-based datasets (StrategyQA, MuSR, Math-lvl-5, SQuAD 2.0).
The results indicate that consensus protocols excel in knowledge-intensive domains while voting and judge protocols are more effective for logic-based tasks. Increasing response diversity through independent solution generation improves decision quality, while changes in information access during the decision process have minimal impact.
Blogger's Review: This paper introduces a novel perspective on the application of large language models by implementing a multi-agent framework and various decision protocols. The strong performance of consensus protocols in knowledge-intensive tasks highlights the potential of multi-agent systems in solving complex problems, warranting further exploration and practical application.