# CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation

Authors: Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, Yulan He.

Uses self-distillation to align continuous internal reasoning states with explicit chain-of-thought behavior while reducing visible reasoning tokens.

First public: 2025-02-28.
ArXiv initial submission: 2025-02-28.

[Original paper](https://arxiv.org/abs/2502.21074) · [Reading-list entry](https://latent-implicit-thinking.github.io/papers/2502.21074/)
[Preprint BibTeX](https://latent-implicit-thinking.github.io/papers/2502.21074/citation.bib) · [Preprint CSL-JSON](https://latent-implicit-thinking.github.io/papers/2502.21074/citation.csl.json) · [JSON record](https://latent-implicit-thinking.github.io/papers/2502.21074/record.json)

## Method comparisons

**Hidden-state distillation: the 2023 method and CODI**

The 2023 implicit CoT paper proposes reasoning through hidden states and implements it with a teacher, a thought emulator and an answer-generating student. CODI also transfers explicit reasoning through hidden-state supervision, but uses one model for both teacher and student tasks. It aligns their states just before the answer, while the student takes successive continuous reasoning steps.

Sources: [Deng et al. (2023), §2.3](https://arxiv.org/html/2311.01460v1#S2.SS3) · [Deng et al. (2023), §3](https://arxiv.org/html/2311.01460v1#S3) · [CODI, §3](https://arxiv.org/html/2502.21074v3#S3)

Paper details: [Implicit Chain of Thought Reasoning via Knowledge Distillation](https://latent-implicit-thinking.github.io/papers/2311.01460/) · [CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation](https://latent-implicit-thinking.github.io/papers/2502.21074/)

**Coconut and CODI: similar computation, different training**

Coconut and CODI both pass continuous states between reasoning steps, but supervise them differently. Coconut gradually replaces written steps with hidden-state inputs and trains on the remaining text. CODI jointly trains explicit and implicit reasoning tasks, matching their hidden states at a selected position before the answer. These models avoid decoding a word at every intermediate step; they still perform successive model passes, so fewer written tokens does not mean no reasoning-time computation.

Sources: [Coconut, §3](https://arxiv.org/html/2412.06769v3#S3) · [CODI, §3](https://arxiv.org/html/2502.21074v3#S3)

Paper details: [Training Large Language Models to Reason in a Continuous Latent Space](https://latent-implicit-thinking.github.io/papers/2412.06769/) · [CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation](https://latent-implicit-thinking.github.io/papers/2502.21074/)


## Published version

CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation. Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, Yulan He. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025.
[Publication](https://aclanthology.org/2025.emnlp-main.36/) · [BibTeX](https://latent-implicit-thinking.github.io/papers/2502.21074/publication.bib) · [CSL-JSON](https://latent-implicit-thinking.github.io/papers/2502.21074/publication.csl.json)
Metadata checked 2026-10-18 against [the publication record](https://aclanthology.org/2025.emnlp-main.36/).

## Approaches

- [Internalized reasoning](https://latent-implicit-thinking.github.io/topics/internalized-reasoning/): Learn from worked solutions so that some or all of the reasoning can later happen without writing those steps. Training may use a separate teacher model or progressively remove written steps from the model being trained.
- [Continuous-state reasoning](https://latent-implicit-thinking.github.io/topics/continuous-states/): Use vectors—lists of numbers—to carry intermediate reasoning from one step to the next, instead of selecting a text token at every step. Some methods use the model's hidden states; others combine several candidate token representations.

Summary and categories are editorial. Read the original paper to verify claims and limitations. A citation to a paper does not, by itself, establish that a later method builds on it.

Reading-list version 0.9.5; literature checked through 2026-10-18.