We investigate aligning two independently pretrained unimodal encoders when only $N \sim 100$--$500$ paired examples are available while abundant unpaired data exists on each side. Each representation is expressed by its similarities to the $N$ paired anchors, embedding both modalities into a shared coordinate system. A single entropic optimal transport problem is then solved over the whole pool; its marginal constraints counteract the hubness that hampers nearest‑neighbor matching. The resulting transport plan is distilled into two lightweight encoders, rendering the correspondence inductive. We also relate this marginal constraint to the one‑step hub corrections used in bilingual lexicon induction. Pairing is required solely for the anchors; the unpaired collections can be drawn independently from the two marginals, so no item outside the anchor set needs both modalities observed. Evaluations on image‑text and molecule‑text alignment tasks—Flickr30k, MS‑COCO, and ChEBI‑20—show that our method outperforms existing approaches given the same anchor budget.
Review