Think before you speak: Training language models with pause tokens
Introduces pause tokens that give a language model additional computational positions before it must produce an answer.
First public:
Cite this paper
Published version
Think before you speak: Training Language Models With Pause Tokens
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, Vaishnavh Nagarajan
International Conference on Learning Representations, 2024, pp. 27896-27923.
Checked 2026-10-18 against the publication record. The published title, author list or year can differ from the preprint.
Choose the version you used. The arXiv link may serve a revision; check the original paper for the exact result, assumptions and limitations you cite.
Method comparisons
Pausing and planning
Pause tokens add computation before the answer without choosing a different plan token for each step. Planning tokens instead select a learned token before generating each written reasoning step. The latter still produces a textual chain of thought. In the pause-token experiments, the training setup matters: gains were strongest when pauses were used in both pretraining and fine-tuning, not simply inserted into an unchanged model.
Sources: Pause-token paper · Planning-token paper, §2
Paper details: Think before you speak: Training language models with pause tokens · Guiding Language Model Reasoning with Planning Tokens
Explore this approach
Thinking tokens — Give a model extra tokens for computation, such as pauses, filler symbols, or learned planning tokens. These tokens need not spell out the solution.
Read or reuse this entry
Plain-text Markdown · JSON record · Find in the full list
Our summary and categories are a reading aid, not the authors’ abstract. Use the original paper for research claims. Suggest a correction.