Weight‑space learning faces a major obstacle when moving from task‑specific or vision models to large language models (LLMs): the lack of a curated, analyzable population of pretrained LLMs. To fill this gap we built a model zoo by varying the data‑mixture composition—web text, books, mathematics, code, and French—across three scales of GPT‑2 (51M, 124M, 355M parameters), training 100 models per scale (N=100). We evaluate the populations with cosine similarity, functional diversity, leave‑one‑mixture‑out experiments, and variance decomposition, showing that model variation is structured by the training mixture. The mixture composition is embedded as a conditioning vector for a Flow Matching model operating in Gram‑PCA‑compressed weight space, learning to generate full pretrained weights. Zero‑shot tests on held‑out mixtures achieve high Continuous Ranked Probability Score (CRPS), while downstream perplexity and benchmark scores change minimally, indicating that the generator can synthesize models ex nihilo that match the zoo’s performance. All code and artifacts are released to support further expansion of weight‑space learning for LLMs.
Review: This study proves that data‑conditioned flow matching can reliably generate high‑quality LLM weights, opening a systematic path for weight‑space research in the LLM era.