We present AV-JEPA, an elegant multimodal extension of LeJEPA focused on audio-visual self-supervised learning. The model employs an early-fusion Vision Transformer and modality dropout as masking, aiming to align the embeddings of global and per-modality local views, while the SIGReg objective encourages a theoretically optimal distribution.
This approach achieves cross-modal alignment in the latent space, resulting in a remarkably clean architecture with no decoder, EMA teacher, complex multi-term losses, or contrastive negatives. The proposed AV-JEPA backbone delivers competitive classification performance on VGGSound (57.1% top-1) and AudioSet (32.7 mAP) and supports zero-shot audio-video retrieval out of the box.
Blogger's Review: AV-JEPA demonstrates the immense potential of audio-visual self-supervised learning by simplifying architecture and achieving effective modality alignment, especially providing efficient performance in practical applications.