LoRi: Low-Rank Distillation for Implicit Reasoning
Distills a teacher's written reasoning into fewer continuous steps by matching hidden-state means and covariances in a shared low-rank subspace. Adds a separate alignment loss where latent reasoning hands off to answer generation.
First public: · Added to this list: November 2, 2026
Explore this approach
Internalized reasoning — Learn from worked solutions so that some or all of the reasoning can later happen without writing those steps. Training may use a separate teacher model or progressively remove written steps from the model being trained.
Continuous-state reasoning — Use vectors—lists of numbers—to carry intermediate reasoning from one step to the next, instead of selecting a text token at every step. Some methods use the model's hidden states; others combine several candidate token representations.