NeFut Logo NeFut
中 Admin Login

[CS.AI] Leaner Training, Lower Leakage: Revisiting Memorization in LLM Fine-Tuning with LoRA

Published at: 2026-11-04 22:00 Last updated: 2026-11-05 00:45
#Machine Learning #optimization #LLM

Memorization in large language models creates vulnerability to data‑extraction attacks. While pre‑training memorization has been extensively studied, the fine‑tuning stage—especially the parameter‑efficient LoRA approach—has received far less attention. This work re‑examines memorization during fine‑tuning and uncovers a striking divergence from earlier findings across different fine‑tuning strategies. Factors such as model scale and data duplication, which amplify memorization in pre‑training and full‑parameter fine‑tuning, do not follow the same pattern in LoRA fine‑tuning. Using a more relaxed similarity‑based memorization metric, experiments show that LoRA markedly reduces memorization risk compared with full fine‑tuning while preserving strong task performance.\n\nReview: By constraining the trainable sub‑space, LoRA weakens excessive data memorization without sacrificing downstream accuracy, offering a practical route toward safer and more efficient LLM deployment.

Original Source: https://arxiv.org/abs/2506.20856

[h] Back to Home