CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation
Uses self-distillation to align continuous internal reasoning states with explicit chain-of-thought behavior while reducing visible reasoning tokens.
First public:
Cite this paper
Published version
CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation
Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, Yulan He
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 677-693.
DOI: 10.18653/v1/2025.emnlp-main.36
Checked 2026-10-18 against the publication record. The published title, author list or year can differ from the preprint.
Choose the version you used. The arXiv link may serve a revision; check the original paper for the exact result, assumptions and limitations you cite.
Method comparisons
Hidden-state distillation: the 2023 method and CODI
The 2023 implicit CoT paper proposes reasoning through hidden states and implements it with a teacher, a thought emulator and an answer-generating student. CODI also transfers explicit reasoning through hidden-state supervision, but uses one model for both teacher and student tasks. It aligns their states just before the answer, while the student takes successive continuous reasoning steps.
Sources: Deng et al. (2023), §2.3 · Deng et al. (2023), §3 · CODI, §3
Paper details: Implicit Chain of Thought Reasoning via Knowledge Distillation · CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation
Coconut and CODI: similar computation, different training
Coconut and CODI both pass continuous states between reasoning steps, but supervise them differently. Coconut gradually replaces written steps with hidden-state inputs and trains on the remaining text. CODI jointly trains explicit and implicit reasoning tasks, matching their hidden states at a selected position before the answer. These models avoid decoding a word at every intermediate step; they still perform successive model passes, so fewer written tokens does not mean no reasoning-time computation.
Sources: Coconut, §3 · CODI, §3
Paper details: Training Large Language Models to Reason in a Continuous Latent Space · CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation
Explore this approach
Continuous-state reasoning — Use vectors—lists of numbers—to carry intermediate reasoning from one step to the next, instead of selecting a text token at every step. Some methods use the model's hidden states; others combine several candidate token representations.
Internalized reasoning — Learn from worked solutions so that some or all of the reasoning can later happen without writing those steps. Training may use a separate teacher model or progressively remove written steps from the model being trained.
Read or reuse this entry
Plain-text Markdown · JSON record · Find in the full list
Our summary and categories are a reading aid, not the authors’ abstract. Use the original paper for research claims. Suggest a correction.