As health AI evolves from simple question-answering systems to agentic systems that can converse with patients, reason about health records, and act on their behalf, primary care plays a critical role in preventing diagnostic errors and unsafe care. Thus, evaluating agents in this domain against these risks is essential. Current benchmarks primarily focus on medical knowledge, assessed through isolated question-answering or clinician-facing tasks.\n\nPatientAgentBench benchmarks patient-facing agentic healthcare; it evaluates a foundation model wrapped in an agent with a sandbox of healthcare tools, conversing with a simulated patient. Each conversation is scored by an LLM-as-a-Jury across six dimensions using over a hundred conversation-agnostic, clinician-grounded criteria. To validate alignment, licensed clinicians annotated shared conversations, achieving 79-93% adjacent agreement between jury and expert raters, comparable to or exceeding clinician inter-rater agreement.\n\nWe benchmarked 10 models across four families on 1,200 scenarios and identified clinical gaps. Triage quality is the most discriminating dimension: pass rates rise from 32% for the weakest models to 88% for the strongest, with agents often acting on administrative requests without clinical screening. Clinical safety and workflow accuracy follow the same trend: the weakest models often fail, fabricating unexecuted actions, while frontier models fail only 1-3% of the time, due to unverified tool outputs and omitted crisis resources in emergencies. More capable models narrow these gaps but do not close them; the strongest scores only 4.25 out of 5 overall. These failures surface only in sustained, tool-using conversations against realistic patient records, confirming that static benchmarks are insufficient as healthcare agentic systems gain autonomy. We release the framework as a reproducible, clinician-validated evaluation standard to help the field close this gap.\n\nBlogger's Review: PatientAgentBench provides a systematic framework for evaluating health AI, emphasizing the importance of clinical safety. As AI agents become more autonomous, traditional evaluation methods prove inadequate. The release of this framework will help advance AI in healthcare, improving patient safety and service quality.