Large language models (LLMs) are increasingly deployed as automatic judges to assess chatbots and other AI systems. In real‑world use, the alignment between the LLM and the intended human evaluation policy can degrade as user interactions and score distributions shift. Existing prompt‑optimization techniques are mostly one‑shot on static development sets, offering limited guidance for long‑term maintenance.\n\nWe term this phenomenon judge drift and introduce PromptOps, a closed‑loop framework for monitoring and iteratively recalibrating LLM‑as‑a‑Judge (LLMaaJ) systems. PromptOps separates the maintenance workflow into four decisions: when to optimize, which optimization algorithm to apply, which feedback records to use, and how to update the judge prompt.\n\nWe instantiate the framework with five prompt‑optimization methods: GEPA, SIMBA, Rubric Synthesis, and exemplar selection via either simulated annealing or a Tree‑structured Parzen Estimator (TPE). Experiments show that iterative prompt optimization can recalibrate LLM judges and recover performance after controlled shifts in the reference‑score distribution. However, effective maintenance is highly context‑dependent: no single algorithm or update schedule generalizes across all settings.\n\nThese findings establish prompt optimization for deployed LLM judges as an inherently iterative maintenance problem and demonstrate that PromptOps offers a practical abstraction for studying and implementing this process.\n\nReview: PromptOps shifts prompt optimization from a static, one‑off experiment to a continuous feedback loop, highlighting the prevalence of judge drift in production and the need for context‑aware maintenance strategies. It provides a clear roadmap for building more robust LLM‑based evaluation pipelines.