Recent studies have explored replacing discrete tokens with continuous representations, creating what is called soft Chain‑of‑Thought (CoT) reasoning. The key idea is that a mixture of token embeddings can encode a superposition of multiple reasoning paths in a single step. However, training large language models to exploit this superposition remains difficult. Existing approaches treat soft tokens as actions in a Markov decision process and apply reinforcement‑learning style training, which prevents gradients from flowing across soft tokens and ignores how one soft token influences the next.
This paper introduces SpaRLax, a sparse relaxation method for reinforcement learning. It treats soft‑CoT reasoning as differentiable computation over time steps and employs an entmax sparsity operator that adaptively determines the degree of softness. Entmax truncates long‑tail noise when the model is confident, yielding sparse outputs, while preserving multiple plausible candidates under uncertainty to maintain reasoning diversity.
Across nine reasoning benchmarks, SpaRLax achieves the highest average performance among all baselines. Additional analyses confirm that the superposition of soft tokens is effectively leveraged and that adaptive sparsity contributes significantly to the gains.
Review