As AI agents increasingly depend on external tools, retrieval accuracy becomes a cornerstone of reliable behavior. Existing approaches focus on semantic relevance while largely overlooking tool availability during execution. To fill this gap, we introduce RATED (Reliability‑Aware Tool Retrieval via Execution Dynamics), a framework that augments semantic matching with dynamic reliability adaptation.
RATED uses a frozen dual‑encoder architecture fine‑tuned with contrastive learning to obtain strong semantic matches. The retrieved candidates are then re‑ranked by a lightweight contextual bandit module that incorporates real‑time execution feedback, enabling rapid adaptation to tool failures without costly model retraining.
We release EcoAgent, a benchmark comprising 5,172 real‑world tools and over 210,000 query‑tool pairs for offline retrieval and dynamic reliability evaluation. Experiments show that RATED achieves a 52.2% relative MRR improvement over the ToolLLM API retriever on real queries, and reduces the fault selection rate from 86.2% to 2.9% under severe outage scenarios with a 30% failure rate, while adding less than 1 ms latency compared to encode‑only inference.
The code and dataset are available at https://github.com/Elliotes/EcoAgent.
Review: By feeding execution‑time reliability signals back into the retrieval loop, RATED demonstrates a practical route to enhance the robustness of AI agents in production settings.