# Efficient Pretraining Length Scaling

Authors: Bohong Wu, Shen Yan, Sijun Zhang, Jianqiao Lu, Yutao Zeng, Ya Wang, Xun Zhou.

Adds hidden decoding tokens during pretraining, then discards their key/value cache entries after use. Keeps the original tokens' entries to preserve longer-range context.

First public: 2025-04-21.
ArXiv initial submission: 2025-04-21.

[Original paper](https://arxiv.org/abs/2504.14992) · [Reading-list entry](https://latent-implicit-thinking.github.io/papers/2504.14992/)
[Preprint BibTeX](https://latent-implicit-thinking.github.io/papers/2504.14992/citation.bib) · [Preprint CSL-JSON](https://latent-implicit-thinking.github.io/papers/2504.14992/citation.csl.json) · [JSON record](https://latent-implicit-thinking.github.io/papers/2504.14992/record.json)

## Approaches

- [Thinking tokens](https://latent-implicit-thinking.github.io/topics/thinking-tokens/): Give a model extra tokens for computation, such as pauses, filler symbols, or learned planning tokens. These tokens need not spell out the solution.
- [Recurrent reasoning](https://latent-implicit-thinking.github.io/topics/recurrent-reasoning/): Repeat parts of the model or keep intermediate information in memory while working through a problem. Some methods also learn how many passes to run.

Summary and categories are editorial. Read the original paper to verify claims and limitations. A citation to a paper does not, by itself, establish that a later method builds on it.

Reading-list version 0.9.5; literature checked through 2026-10-18.