This paper explores the potential for poisoning pretraining data, which can lead to harmful behaviors in language models (LMs) that are difficult to detect and mitigate. Previous work has largely focused on established data sources like Wikipedia, which do not represent the large scale and heterogeneity typical of pretraining corpora, and has overlooked the interaction between poisoned data and data curation pipelines.
We demonstrate that poisoning attacks on pretraining data are feasible beyond this limited setting through an existing web-scale content injection mechanism: public discussion interfaces. Additionally, to measure whether malicious content is included after web crawling and data curation, we introduce HalfLife, a novel analysis for estimating adversarial content inclusion in web-crawl based LM training data.
Using HalfLife, we explore the feasibility of poisoning pretraining corpora at web scale through open discussion interfaces. Our analysis underscores the importance of estimating whether poison injections are included in pretraining data and establishes third-party webpage content as a possible vector for attacking language model pretraining.
Blogger's Review: This research highlights the severity of pretraining data poisoning, especially in environments with open discussion platforms. As language models become more widely used, ensuring the security and reliability of training data is crucial, necessitating greater attention and technological measures to mitigate this potential threat.