Spiking neural networks attract interest for their brain‑inspired dynamics, yet native large‑scale spiking language‑model pre‑training remains largely untapped. We identify a key bottleneck: standard spiking neurons use a uniform firing sensitivity for every token, producing either dense or silent firing regimes that a single global parameter set cannot reconcile with the highly variable informational roles of tokens in a sentence. To overcome this, we introduce NeuronSpark, a self‑attention‑free language model built around input‑dependent spiking membrane dynamics. Each neuron’s decay rate, gain, and firing threshold adapt to the current token content. Coupled with gated associative memory and adaptive temporal aggregation, the model can be pretrained from random initialization at 1.1 B parameters and 2.1 B tokens without backbone transfer, conversion, or distillation. We relate the neuron to fixed‑parameter spiking models and the memory to linear attention. Experiments show NeuronSpark achieves a 40.0 average on eight zero‑shot tasks, surpassing comparable‑budget Pythia and Mamba checkpoints while retaining 95.2%/93.2% of their fully‑trained scores and spontaneously forming a fast‑slow hierarchy. At a sequence length of 2048, a parameter‑matched VHDL implementation reduces operation‑energy proxy by 90.1%/67.0% and cycles by 94.1%/80.3% compared to Transformer/Mamba. A physical FPGA prototype reaches 24.000 s TTFT and 0.082440 TPS throughput.
Review