# Implicit Chain of Thought Reasoning via Knowledge Distillation

Authors: Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, Stuart Shieber.

First proposes implicit chain-of-thought reasoning: language models carry out intermediate reasoning in continuous hidden states rather than generate it as text. Defines this setting separately from its implementation, which distills an explicit-CoT teacher's hidden states into an emulator and a student that produces the answer.

First public: 2023-11-02.
ArXiv initial submission: 2023-11-02.

[Original paper](https://arxiv.org/abs/2311.01460) · [Reading-list entry](https://latent-implicit-thinking.github.io/papers/2311.01460/)
[Preprint BibTeX](https://latent-implicit-thinking.github.io/papers/2311.01460/citation.bib) · [Preprint CSL-JSON](https://latent-implicit-thinking.github.io/papers/2311.01460/citation.csl.json) · [JSON record](https://latent-implicit-thinking.github.io/papers/2311.01460/record.json)
[Code](https://github.com/da03/implicit_chain_of_thought/)

Related terms: implicit CoT, latent reasoning, latent chain-of-thought, latent CoT.

## Method comparisons

**Distillation and stepwise internalization**

Deng et al. (2023) introduced implicit CoT as reasoning through hidden states rather than written steps; distillation is the paper's implementation of that proposal. It learns to predict a teacher's intermediate hidden states, which help a student produce the answer. Stepwise Internalization instead repeatedly shortens the written solution and fine-tunes the same model on what remains. Both use worked solutions during training; neither requires writing the removed steps at inference. Their reported experiments focus on multiplication and grade-school math, not a guarantee of general reasoning ability.

Sources: [Implicit CoT proposal, §2.3](https://arxiv.org/html/2311.01460v1#S2.SS3) · [Distillation implementation, §3](https://arxiv.org/html/2311.01460v1#S3) · [Stepwise internalization, §3](https://arxiv.org/html/2405.14838v1#S3)

Paper details: [Implicit Chain of Thought Reasoning via Knowledge Distillation](https://latent-implicit-thinking.github.io/papers/2311.01460/) · [From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step](https://latent-implicit-thinking.github.io/papers/2405.14838/)

**Hidden-state distillation: the 2023 method and CODI**

The 2023 implicit CoT paper proposes reasoning through hidden states and implements it with a teacher, a thought emulator and an answer-generating student. CODI also transfers explicit reasoning through hidden-state supervision, but uses one model for both teacher and student tasks. It aligns their states just before the answer, while the student takes successive continuous reasoning steps.

Sources: [Deng et al. (2023), §2.3](https://arxiv.org/html/2311.01460v1#S2.SS3) · [Deng et al. (2023), §3](https://arxiv.org/html/2311.01460v1#S3) · [CODI, §3](https://arxiv.org/html/2502.21074v3#S3)

Paper details: [Implicit Chain of Thought Reasoning via Knowledge Distillation](https://latent-implicit-thinking.github.io/papers/2311.01460/) · [CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation](https://latent-implicit-thinking.github.io/papers/2502.21074/)

**Implicit CoT: proposal and implementation**

Deng et al. (2023) proposed implicit chain-of-thought reasoning: language models carry out intermediate reasoning in continuous hidden states rather than generate it as text. Section 2.3 defines this setting; section 3 develops a distillation method to implement it. That method transfers a teacher's reasoning into hidden states across layers. Reasoning through continuous hidden states is the broader proposal; it does not require the particular distillation procedure used in this paper. Its across-layer computation differs from the repeated hidden-state feedback used by later methods such as Coconut.

Sources: [Implicit CoT proposal, §2.3](https://arxiv.org/html/2311.01460v1#S2.SS3) · [Distillation implementation, §3](https://arxiv.org/html/2311.01460v1#S3) · [Coconut, §3](https://arxiv.org/html/2412.06769v3#S3)

Paper details: [Implicit Chain of Thought Reasoning via Knowledge Distillation](https://latent-implicit-thinking.github.io/papers/2311.01460/) · [Training Large Language Models to Reason in a Continuous Latent Space](https://latent-implicit-thinking.github.io/papers/2412.06769/)


## Demos

Related-method demo: [Multiplication without written steps](https://huggingface.co/spaces/yuntian-deng/gpt2-multiplication)

Compare GPT-2 models using implicit CoT, explicit CoT, or no CoT on two numbers of up to 20 digits each.

The models can make mistakes, including on valid inputs. The implicit model writes answer digits, not intermediate steps.

[Model details](https://huggingface.co/yuntian-deng/gpt2-implicit-cot-multiplication-20-digits)

Implements: [From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step](https://latent-implicit-thinking.github.io/papers/2405.14838/).

Related method, not implemented by this demo: [Implicit Chain of Thought Reasoning via Knowledge Distillation](https://latent-implicit-thinking.github.io/papers/2311.01460/).

Related-method demo: [Math word problems without written steps](https://huggingface.co/spaces/yuntian-deng/implicit-cot-math)

Try a Mistral-7B model trained with stepwise internalization on grade-school math word problems.

Answers can be wrong. This model is trained for grade-school math word problems, not general conversation. It still generates the answer token by token.

[Model details](https://huggingface.co/yuntian-deng/implicit-cot-math-mistral7b)

Implements: [From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step](https://latent-implicit-thinking.github.io/papers/2405.14838/).

Related method, not implemented by this demo: [Implicit Chain of Thought Reasoning via Knowledge Distillation](https://latent-implicit-thinking.github.io/papers/2311.01460/).


## Approaches

- [Internalized reasoning](https://latent-implicit-thinking.github.io/topics/internalized-reasoning/): Learn from worked solutions so that some or all of the reasoning can later happen without writing those steps. Training may use a separate teacher model or progressively remove written steps from the model being trained.

Summary and categories are editorial. Read the original paper to verify claims and limitations. A citation to a paper does not, by itself, establish that a later method builds on it.

Reading-list version 0.9.4; literature checked through 2026-10-18.