Large language models are now trained and evaluated under a variety of paradigms: supervised fine‑tuning (SFT), few‑shot in‑context learning (ICL), KL‑regularized RLHF/RLVR, on‑policy distillation (OPD), and test‑time search with chain‑of‑thought. These methods appear fundamentally different, and mixed empirical findings—such as the inconsistent effect of few‑shot prompting on RL‑tuned reasoning models—can be puzzling. This note adopts a Bayesian perspective that places all of them on equal footing. The central two‑step template is: (i) given a context, construct a (generalized) Bayes or Gibbs posterior $q^{*}$ using a prior/reference model and a utility signal (log‑likelihood, reward, or advantage); (ii) approximate $q^{*}$ by a forward‑KL projection onto a parametric family, either in weight space (SFT/RL) or in‑context space (ICL).
Part I formalizes few‑shot ICL and SFT as amortized or in‑context KL projections onto the Bayes posterior predictive. Parts II‑IV show that KL‑regularized RLHF/RLVR, reward‑weighted SFT, reward‑weighted ICL (RW‑ICL), and advantage‑weighted SFT (AWSFT) are all instances of forward‑KL projection onto posteriors induced by rewards or advantages. We disentangle where these equivalences hold—objectives and first‑order updates—and where they break—source and granularity of the learning signal.
Part V sketches implications for modern reasoning pipelines: RLHF/RLVR can be seen as “posterior design + projection”, importance‑weighted KL projections practically require a cold‑start or supervised warm‑up, and models such as DeepSeek‑R1 and o1‑style reasoning systems combine test‑time Bayesian search with training‑time KL amortization.
Review