Large language models (LLMs) have attracted extensive attention in both academia and industry in recent years. Existing surveys have covered learning strategies, applications, alignment, explainability, and evaluation, yet a systematic review of data‑related aspects remains lacking. Data is recognized as a key factor driving the LLM era, and a wide range of data optimization algorithms are employed during training and inference. These algorithms originate from diverse deep‑learning subfields, and their theoretical inspirations and heuristic motivations often appear unrelated.
This paper proposes a unified framework for LLM data optimization to improve data utilization and efficiency. First, it outlines the data challenges encountered in LLM training and inference. Then it introduces perception dimensions for evaluating data properties—quality, difficulty, diversity, and redundancy—providing a foundation for subsequent optimization. Existing techniques are categorized into four groups: data selection, data augmentation, data reweighting, and other methods, and the interconnections among these groups are analyzed. Finally, by comparing with conventional deep learning, the paper identifies key challenges and outlines future directions for LLM data optimization.
Review