Representation‑based safety defenses usually learn to separate harmful from benign inputs, yet separation alone does not clarify what a risk score means nor when it can be trusted. We introduce Counterfactual Neighborhood Risk Fields (CNRF), an evidence‑grounded framework for selective safety of vision‑language models. CNRF first builds context‑matched benign‑harmful counterfactual pairs and uses their representation differences to define risk directions tied to model capability. It then estimates neighborhood support separately from risk, preventing unfamiliarity from being conflated with harmfulness and enabling selective intervention instead of blanket rejection. A victim‑adaptive interface accommodates different safety observations across VLM architectures and routes selected requests to a same‑victim response review module, which either releases acceptable candidates or produces constrained alternatives with reduced harmful capability. Experiments across three VLMs and five multimodal jailbreak families show a macro defense success rate of 86.06%‑89.30% while retaining substantial benign‑task utility. Controlled studies reveal that query‑matched references and preserved context‑direction correspondence materially improve risk ranking in the low‑false‑positive regime. CNRF thus shifts representation‑based VLM safety from learning harmfulness boundaries to constructing evidence for selective risk decisions.
Review