We build a vision‑language reward model without any human preference labels. The key is to attach a linear head to an intermediate layer of a frozen Qwen3‑VL backbone and train it to score the original web caption higher than a perturbed version of the same caption; labeled data is only used to pick the layer. The layer choice proves decisive, giving a 5.8‑point boost over the final layer. With this setup, a few thousand parameters at roughly one‑thousandth the fine‑tuning compute achieve performance that matches or exceeds GPT‑4o on three benchmarks at 32B scale, and lead by 6.6 points on VL‑RewardBench.
When the pair budget is equal, perturbation‑based training of a frozen head outperforms curated RLAIF‑V pairs by 3‑5 points. Because the backbone is frozen, the objective is convex and the perturbation sampler itself defines the reward, something the benchmark cannot capture: two heads trained on disjoint halves of the same perturbations tie on the benchmark yet differ by 6.4 points in hallucination judging.
Retraining the head on verifier‑labeled candidates generated by the policy reduces caption hallucination by 5.1 CHAIR points and raises image‑generation pass rate by 5.4 points, even though it scores lower on every reward benchmark.
Together, an intermediate‑layer probe, free same‑image perturbations, and a verifier‑driven bootstrap provide a recipe for multimodal reward modeling that requires no annotators, costs a fraction of fine‑tuning, and is specified by the comparisons it is trained on rather than by a benchmark rank.
Review