# Think before you speak: Training language models with pause tokens

Authors: Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, Vaishnavh Nagarajan.

Introduces pause tokens that give a language model additional computational positions before it must produce an answer.

First public: 2023-10-03.
ArXiv initial submission: 2023-10-03.

[Original paper](https://arxiv.org/abs/2310.02226) · [Reading-list entry](https://latent-implicit-thinking.github.io/papers/2310.02226/)
[Preprint BibTeX](https://latent-implicit-thinking.github.io/papers/2310.02226/citation.bib) · [Preprint CSL-JSON](https://latent-implicit-thinking.github.io/papers/2310.02226/citation.csl.json) · [JSON record](https://latent-implicit-thinking.github.io/papers/2310.02226/record.json)

## Method comparisons

**Pausing and planning**

Pause tokens add computation before the answer without choosing a different plan token for each step. Planning tokens instead select a learned token before generating each written reasoning step. The latter still produces a textual chain of thought. In the pause-token experiments, the training setup matters: gains were strongest when pauses were used in both pretraining and fine-tuning, not simply inserted into an unchanged model.

Sources: [Pause-token paper](https://proceedings.iclr.cc/paper_files/paper/2024/hash/76917808731dae9e6d62c2a7a6afb542-Abstract-Conference.html) · [Planning-token paper, §2](https://arxiv.org/html/2310.05707v3#S2)

Paper details: [Think before you speak: Training language models with pause tokens](https://latent-implicit-thinking.github.io/papers/2310.02226/) · [Guiding Language Model Reasoning with Planning Tokens](https://latent-implicit-thinking.github.io/papers/2310.05707/)


## Published version

Think before you speak: Training Language Models With Pause Tokens. Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, Vaishnavh Nagarajan. International Conference on Learning Representations, 2024.
[Publication](https://proceedings.iclr.cc/paper_files/paper/2024/hash/76917808731dae9e6d62c2a7a6afb542-Abstract-Conference.html) · [BibTeX](https://latent-implicit-thinking.github.io/papers/2310.02226/publication.bib) · [CSL-JSON](https://latent-implicit-thinking.github.io/papers/2310.02226/publication.csl.json)
Metadata checked 2026-10-18 against [the publication record](https://proceedings.iclr.cc/paper_files/paper/2024/hash/76917808731dae9e6d62c2a7a6afb542-Abstract-Conference.html).

## Approaches

- [Thinking tokens](https://latent-implicit-thinking.github.io/topics/thinking-tokens/): Give a model extra tokens for computation, such as pauses, filler symbols, or learned planning tokens. These tokens need not spell out the solution.

Summary and categories are editorial. Read the original paper to verify claims and limitations. A citation to a paper does not, by itself, establish that a later method builds on it.

Reading-list version 0.9.5; literature checked through 2026-10-18.