Scaling laws state that a language model becomes more capable as its parameter count and training data grow, a principle that Mixture‑of‑Experts (MoE) architectures exploit by activating only a tiny fraction of a massive parameter bank for each token. This success relies on static pre‑training data. In deployment, a model encounters a different world where many useful facts and corrections come from live interaction and are absent from its training set. Conventional models cannot learn from such data because their weights are frozen after training, so knowledge and behavior are injected via prompts through retrieval or instruction and discarded after each request. We ask how an architecture could write this live interaction into its weights. Inspired by MoE, we propose the Infinite‑Parameter LLM. A compact hypernetwork turns run‑time data into a low‑rank modulation of a shared base network, so feed‑forward weights are generated from live data rather than stored in a fixed bank. Unlike prior weight generators that read context once and freeze, we maintain a Bayesian belief over the generator’s latent code and update it online; the effective weight is re‑derived from this evolving belief as the session proceeds. The stored footprint stays constant, yet the compile‑time weights are effectively infinite. Storing run‑time knowledge in weights instead of prompts amortizes compute, frees the context window, persists across turns, and can generalize better than in‑context use. We define an evaluation protocol that tests this approach directly against in‑context learning and retrieval.
Review