Before an AI agent accepts a paid gig, its likelihood of success must be judged. We introduce PAYWORTHY, a dataset of 1,494 real freelance postings sampled from 18.8 M listings between 2020 and 2026. The experimental pipeline is:
- The agent attempts each job using only the posting text;
- An LLM acting as the client decides whether to accept the work;
- Eighteen frontier language models forecast the agent’s success probability, and we evaluate how well the forecasts rank accepted jobs above rejected ones using AUC (0.5 = random ranking).
When the agent is described with explicit limitations, models achieve AUCs of 0.66‑0.86, indicating good ranking. When the description merely calls the agent a generic system with a browser and code execution, AUC drops to 0.45‑0.56, comparable to random ordering. These patterns persist across four other judge families (Claude, GLM, Kimi, Qwen) and align with judgments from two human raters.
The implementation is publicly available at https://github.com/upwork/Payworthy.
Review: PAYWORTHY demonstrates that LLMs’ ability to predict an AI agent’s performance hinges on how precisely the agent’s capabilities are framed, suggesting that clear capability specifications are crucial for reliable automated job selection.