Modern large language models (LLMs) are increasingly utilized in long-context applications such as retrieval-augmented generation, repository-level coding, and agentic workflows, where the accumulated reasoning and tool traces often push the input length significantly beyond the pretraining window. Most existing zero-shot methods fix a single rescaling factor upfront, where an aggressive factor sacrifices short-context fidelity, while a conservative one fails at long contexts.
We propose Jet-Long, a tuning-free zero-shot method that combines a local RoPE-faithful window with a long-range window whose rescaling factor dynamically adapts to the current sequence length, accurately recovering the base model at short inputs while extrapolating cleanly at long ones.
An inclusion-exclusion attention merge and on-the-fly RoPE correction rotation make the bifocal construction essentially free at inference; when fused into a single CuTe kernel, long-context prefill achieves up to $1.39\times$ FA2 throughput on H100 (approaching the Hopper-only FA4), with single-batch generation incurring $<4\%$ overhead at every length. On Qwen3-1.7B/4B/8B up to 128K context, Jet-Long outperforms RULER by $+4.79$/$+2.18$/$+2.03$ percentage points over the strongest baseline at 1.7B/4B/8B, achieves the best overall accuracy on HELMET-RAG (identified by HELMET as the most efficient predictor of downstream long-context performance), and attains the lowest PG-19 perplexity.
Jet-Long also generalizes to hybrid attention architectures like Jet-Nemotron for further long-context enhancement without retraining, and remains hyperparameter-resilient for ease of deployment.
Blogger's Review: The Jet-Long method successfully addresses the fidelity-efficiency trade-off in long-context processing by dynamically adjusting the rescaling factor of the long-range window. Its excellent performance across different model scales demonstrates the method's broad applicability and flexibility, making it a noteworthy advancement in LLM applications.