NeFut Logo NeFut
Admin Login

[CS.AI] Whose Record Is This? Diagnosing and Authorizing Record Use in Personalized Multimodal Models

Published at: 2026-09-07 22:00 Last updated: 2026-09-08 00:37
#AI #Machine Learning #LLM

In visual personalization tasks, models may retrieve the correct record but apply it to an incorrect visual subject. We therefore define record authorization, requiring three conditions to hold simultaneously: subject presence ($P$), record‑edge validity ($E$), and answer support ($S$). Violations of any condition lead to visual memory misbinding (VMM).

We built RecordAuth‑Diag, a diagnostic suite of 3,690 matched cases. Each case changes only one image‑record edge while keeping the query, question, record text, and image multiset fixed. Card removal and nonce relabeling experiments attribute these failures primarily to the supplied records.

Raw‑bank failures span Qwen, Phi, and Gemma families. For example, Gemma‑3‑4B‑IT reaches 63.69% local unauthorized use with only 25.75% clean recall. CoViP shows 26.02% unauthorized use, versus 22.49% for its Qwen backbone at comparable clean recall.

Introducing typed pre‑generation authorization reduces Qwen’s card exposure on RecordAuth‑Diag from 43.63% to 3.06%, while positive recall drops from 86.26% to 60.90%. Full $P\wedge E\wedge S$ validation on 560 localized DAVIS cases yields comparable release rates for top‑1 relevance and typed authorization (28.93% and 28.39%) but markedly lower unsafe release (6.79% vs. 0.89%). Among the 33 additional unsafe cases removed, 27 are support errors, 4 edge errors, 2 clean errors, and none are boundary errors. Thus the observed improvement is driven by an $E\wedge S$ decision dominated by support violations rather than edge checks alone.

Appearance provides $E$ evidence only conditional on $P$; authenticated subject tokens instantiate the missing presence witness, acting as a sufficiency control.

Review: The paper formalizes record authorization, quantifies unauthorized usage in multimodal models with a large diagnostic suite, and demonstrates that typed authorization dramatically cuts record leakage while preserving reasonable recall, offering a practical path toward safe visual personalization.

Original Source: https://arxiv.org/abs/2609.04801

[h] Back to Home