Abstract
Continuous Chain-of-Thought (CoT) methods replace verbose reasoning traces with a short sequence of dense latent representations. Earlier continuous CoT methods indirectly supervise the latent representations to ensure their final state matches verbose reasoning traces, requiring autoregressive, slow generation during training.
We introduce C-MTP, a simpler, faster direct supervision approach that models each latent as an average of the embeddings in the CoT traces to be compressed. Our approach outperforms a prior direct supervision method that approximates the distribution of compressed tokens and competes with slower indirect supervision methods in existing evaluation setups with simplified CoT traces (less than 100 tokens).
Lastly, we extend the evaluation of Continuous CoT methods to complex tasks with longer reasoning traces ($\ge$ few hundreds reasoning tokens). We find that both direct and indirect supervision training methods perform poorly (approximately 65\% performance drop) in this setting, revealing the limitations of current continuous CoT methods.
The code and checkpoints are released at GitHub.
Blogger's Review: The C-MTP approach excels in speed and efficiency, yet the significant performance drop in complex tasks highlights the need to balance model simplification with adaptability and generality, which will be a key focus for future research.