Online advertising bidding systems typically rely on multiple offline-trained expert models (e.g., PID controllers, model predictive control, offline RL policies), facing two critical limitations: lack of online adaptability to non-stationary auction markets and reliance on costly manual tuning of hyperparameters such as bid bounds and budget pacing constraints. We propose HOBA (Hierarchical On-policy Bidding Agents), a hierarchical reinforcement learning framework that decouples strategic reasoning, model selection, and bid execution across three time scales.
At the high level, a large language model infers hyperparameters from contextual signals through a Think-Act-Observe-Reflect loop with historical experience retrieval. At the mid level, a SARSA agent dynamically selects among expert models, incorporating causal adjustment to eliminate selection bias. At the low level, a dynamic expert pool (PID, MPC, IQL, Decision Transformer) executes bids under high-level constraints. This design confines online learning to discrete expert selection rather than continuous bid optimization, significantly reducing exploration risk while maintaining adaptability.
Experiments on the AuctionNet benchmark and a large-scale A/B test demonstrate consistent improvements over state-of-the-art baselines. In a large-scale online deployment, HOBA delivered substantial business value, achieving a +3.6% increase in target cost, proving the effectiveness of our hierarchical multi-agent bidding paradigm.
Blogger's Review: The introduction of HOBA presents a novel approach in the online advertising bidding domain, addressing the limitations of traditional methods, particularly in online adaptability and hyperparameter tuning challenges. This innovation has the potential to generate significant economic benefits in practical applications, making it worthy of attention.