Long‑horizon agent interactions generate abundant but noisy experience, and retraining models to absorb it is costly. Consequently, context‑evolving agents need a memory‑extraction method that improves with additional test‑time compute without relying on gold labels. We introduce RefCon, which couples sequential self‑refinement with parallel self‑contrast to extract higher‑quality memories in a label‑free setting.\ \ Evaluations on AppWorld and BFCL‑V3 across several context‑evolving agent frameworks show that RefCon consistently outperforms non‑scaling baselines, delivering relative gains of 21.6% on ACE and 16.6% on ReMe. A diversity‑focused variant, DivCon, achieves a 35.5% improvement on ReasoningBank.\ \ RefCon also generalizes across model scales and to software‑engineering tasks, surpassing even gold‑label baselines. Our analysis of the accuracy‑token trade‑off and scaling behavior reveals that RefCon maintains favorable efficiency and continues to improve as more trajectories are incorporated, whereas diversity‑only scaling saturates earlier.\ \ Review: By integrating self‑refinement and contrastive learning, RefCon markedly enhances memory quality and downstream performance without any human annotation, highlighting its practicality for large‑scale, long‑horizon interaction scenarios.