# LLM Pretraining with Continuous Concepts

Authors: Jihoon Tack, Jack Lanchantin, Jane Yu, Andrew Cohen, Ilia Kulikov, Janice Lan, Shibo Hao, Yuandong Tian, Jason Weston, Xian Li.

Interleaves continuous concept representations with text during pretraining so language models learn to compute over concepts as well as tokens.

First public: 2025-02-12.
ArXiv initial submission: 2025-02-12.

[Original paper](https://arxiv.org/abs/2502.08524) · [Reading-list entry](https://latent-implicit-thinking.github.io/papers/2502.08524/)
[Preprint BibTeX](https://latent-implicit-thinking.github.io/papers/2502.08524/citation.bib) · [Preprint CSL-JSON](https://latent-implicit-thinking.github.io/papers/2502.08524/citation.csl.json) · [JSON record](https://latent-implicit-thinking.github.io/papers/2502.08524/record.json)
[Code](https://github.com/facebookresearch/RAM/tree/main/projects/cocomix)

## Approaches

- [Continuous-state reasoning](https://latent-implicit-thinking.github.io/topics/continuous-states/): Use vectors—lists of numbers—to carry intermediate reasoning from one step to the next, instead of selecting a text token at every step. Some methods use the model's hidden states; others combine several candidate token representations.

Summary and categories are editorial. Read the original paper to verify claims and limitations. A citation to a paper does not, by itself, establish that a later method builds on it.

Reading-list version 0.9.5; literature checked through 2026-10-18.