Cross‑modal translation requires models to capture shared semantics across modalities while preserving the fine‑grained structure demanded by the target modality. In real scientific domains, paired data are often imperfect and obtaining large‑scale clean pairs is costly.
Supervised fine‑tuning (SFT) directly optimizes translation with categorical token‑level supervision, but this objective is highly sensitive to errors in the target sequences. JEPA offers a complementary representation‑space objective, yet most existing formulations focus on prediction within a single modality rather than translation across modalities.
We introduce Anchored Cross‑Modal JEPA (AC‑JEPA), an SFT‑compatible plug‑in that predicts masked intermediate states in both cross‑modal directions using paired data. The joint‑embedding objective targets cross‑modal structure predictable from context, while the causal SFT path retains direct token‑level supervision for the target sequence.
During self‑training, AC‑JEPA learns from continuous latent targets derived from imperfect pseudo‑labeled pairs, providing intrinsic tolerance to local data errors without forcing pseudo‑tokens into hard categorical supervision.
Using optical chemical structure recognition as a testbed, AC‑JEPA trained on clean pairs achieves 76.07% accuracy on the Simple set of the WildMol benchmark, compared with 67.40% for SFT alone. With the same pseudo‑labeled data, AC‑JEPA‑based self‑training further raises accuracy to 80.23%, whereas direct SFT drops to 63.47%.
These results support SFT‑anchored joint‑embedding prediction as an effective strategy for cross‑modal translation under both curated and imperfect supervision.
Review