Silicon-photonic (SiPh) accelerators have emerged as a promising platform for Vision Transformer (ViT) inference through matrix multiplications on microring-resonator (MRR) banks, achieving high throughput and energy efficiency. However, extending these platforms for on-chip fine-tuning is challenging due to the large activation storage, frequent weight write-backs to MRRs, and the need for tolerance to device-level noise in backpropagation.
We present Opto-ViT-v2, the first framework for parameter-efficient fine-tuning (PEFT) on a near-sensor SiPh ViT accelerator. Our tensorized low-rank decomposition separates pretrained optical weights from a small set of trainable electronic factors (as few as 8K parameters for ViT-Base), significantly reducing activation storage and weight updates while enabling practical on-chip training.
We also introduce a gradient-accumulated sparse classifier that freezes low-importance weights using one-shot top-k gradient masking, which reduces classifier training costs by about 40 percent. Additionally, we develop the first system-level noise model for photonic on-chip training, capturing the effects of MRR crosstalk, thermal drift, and laser amplitude noise during both forward and backward propagation.
Calibrated using measurements from over 200 fabricated MRR devices, the model shows that low-rank factor updates are more robust than full fine-tuning and conventional layer-wise low-rank adaptation under identical noise conditions. Experiments on VTAB-1K (19 tasks) and FGVC few-shot benchmarks demonstrate that Opto-ViT-v2 recovers 0.3 to 0.8 percent of clean software accuracy under measured photonic noise while achieving over 100 KFPS/W, facilitating practical on-chip domain adaptation for photonic edge vision systems.