Large language model (LLM) agents can propose, implement, and evaluate model changes. Minute‑scale autoresearch loops demonstrate this capability on a self‑contained program. Online autoresearch, however, spans asynchronous systems, hour‑long variants, and week‑long campaigns that may affect a product. Even a completed run can yield an invalid conclusion when a code change is a no‑op, data windows leak, evaluator semantics drift, or the two arms follow different serving funnels.
We introduce EvoPilot, a human‑gated method for long‑horizon online autoresearch. Role‑specific agents execute each round via a versioned domain skill and typed adapter. Durable records preserve experiments and failures; deterministic checks enforce recorded lessons.
We conducted a 37‑day campaign on the retrieval system powering Video Deep Dive (VDD), an online experience that surfaces follow‑on videos after a user opens a seed video. The campaign used an hourly refreshed index of hundreds of millions of videos. Earlier manual experiments had not shown a benefit from an interaction head. A primitive autoresearch attempt revisited the direction but mistakenly attributed an offline hit‑rate drop of 22 percentage points to the head. After introducing EvoPilot, human‑gated verification traced the drop to a pre‑existing evaluation defect that produced output depths of 3,000 and 600. After fixing the defect, a matched comparison measured an offline improvement of 3.20 percentage points.
Post‑study replay and mutation tests rejected invalid comparisons while admitting valid counterparts. Durable state recovered an interrupted round, and artifact reuse avoided roughly five GPU‑hours. Separately, a seven‑day randomized online evaluation estimated a 0.66 % relative increase in the VDD slice of Good Search Result Rate for Retention (GSRR).
Review