Transductive adaptation refines predictions on an unlabeled batch at test time, improving vision‑language models (VLMs). Existing methods, however, face a fundamental trade‑off: aggressive approaches exploit batch structure but can drift when batches are sparse, exhibit prior shift, or cover only a small part of the label space; conservative approaches preserve CLIP text semantics but may under‑adapt even when the batch offers strong corrective evidence. We introduce JPTA (Joint Prior and Transport Adaptation), a prior‑aware framework that casts transductive VLM adaptation as batch‑conditioned semantic reference estimation. Instead of treating CLIP text prototypes as fixed priors or freely replaceable, JPTA regards them as semantic references that shift toward the image‑side batch structure only under calibrated evidence. Concretely, JPTA first estimates batch priors, then computes soft image‑side prototypes; together they form transported semantic references whose impact is scaled by a batch reliability factor. Across standard benchmarks, low effective‑class regimes, all‑class evaluation, and online streaming scenarios, JPTA consistently outperforms TransCLIP and StatA, with the largest gains on sparse and prior‑shifted batches. Diagnostics—anchor‑drift, active‑support, and absent‑class‑mass—show that JPTA corrects effective‑class mismatch without unconstrained transductive drift, indicating that robust transductive VLM adaptation should recalibrate semantic references using reliable batch evidence rather than choosing between unrestricted transduction and fixed prototypes. The code is released at https://github.com/MR-Sherif/JPTA.git.
Review: JPTA’s batch‑aware prior calibration and soft prototype transport strike a balance between semantic fidelity and data‑driven adaptation, offering a more dependable route for transductive vision‑language model adaptation.