Aggregating predictions from multiple large language models (LLMs) has become increasingly common, particularly when each model possesses specific domain expertise or access to private tools and data. This aggregation enhances collective prediction performance. However, in decentralized settings, determining aggregation weights without access to the models' private information must remain robust against strategic reporting. To address this, we propose a family of advantage-aligned wagering mechanisms for LLM aggregation (WALLA).
In WALLA, each model reports a prediction and a learned wager, with wagers used as weights during aggregation. WALLA introduces a leave-one-out baseline into the net payout function, resulting in three desirable properties:
- Dominant-strategy incentive compatibility under arbitrary belief structures.
- Advantage-wager alignment, where the optimal wager is proportional to the model's expected score advantage.
- Prediction-agnostic wager optimization, enabling decentralized learning of wager policies without requiring optimal predictions.
We further instantiate two mechanism variants that balance normality and no-arbitrage while maintaining a bounded worst-case deficit for the mechanism. Experiments on question-answering and forecasting benchmarks across heterogeneous models and private-information settings demonstrate that WALLA matches centralized aggregation methods in predictive performance while achieving decentralized learning, advantage-aligned aggregation weights, uncertainty awareness, and incentive-compatible predictions.
Blogger's Review: The WALLA mechanism innovatively addresses the issue of information asymmetry in decentralized aggregation by integrating wagering with predictions. This approach not only enhances predictive performance but also fosters trust and collaboration among models, promising significant real-world applications in the future.