Vision-language models (VLMs) can infer sensitive cues such as geolocation from subtle visual hints, which may cause responses to reveal more detail than the image context justifies, creating privacy risks. This work studies visual‑contextual privacy, requiring a VLM to disclose location only at a granularity appropriate to the visual context shown. We first show that even when generated answers are misaligned, the pre‑generation hidden states retain the appropriate granularity, and disclosure levels follow an ordered structure along a common hidden‑state direction. Motivated by this, we introduce an inference‑time route‑and‑steer framework: an ordinal granularity router maps visual‑contextual privacy representations to a disclosure route, and a disclosure‑level‑conditioned steering module intervenes on route‑specific hidden states during decoding. Across four VLM backbones the approach improves alignment between disclosure and visual context, better balances over‑ and under‑disclosure, incurs only modest impact on general capability benchmarks, and transfers to a dataset with a different distribution and privacy‑risk taxonomy.
Review