Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models
Uses mechanistic analysis to identify internal components involved in multi-step reasoning and test their causal roles.
Analysis and evaluation
Measure how well these methods work, investigate what their internal computation does, and identify where they fail. This includes experiments, studies of model internals, and mathematical results.
For example: A researcher replaces an intermediate state with one taken from a different problem, then checks how the answer changes. This tests what information that state contributes to the solution.
How the methods connect and differ
Recovering information and testing its use
Recovering an intermediate answer from hidden states is different from showing that the model uses it. The multi-hop study by Yang and colleagues separates recall of a bridge entity from its contribution to the final answer, including interventions on internal representations. Evidence was stronger for the first hop than for the complete two-hop computation and varied by relation type. That distinction matters when interpreting claims that a model reasons internally.
Sources: Yang et al. (2024)
Paper details: Do Large Language Models Latently Perform Multi-Hop Reasoning?
75 papers, oldest first
Dates usually refer to the first arXiv submission. Search or filter these papers →
A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task
Reverse-engineers a transformer trained on symbolic multi-step reasoning to characterize its learned internal algorithm.
Understanding and Patching Compositional Reasoning in LLMs
Localizes internal representations involved in compositional reasoning and tests whether targeted interventions can repair failures.
Do Large Language Models Latently Perform Multi-Hop Reasoning?
Probes whether language models internally compose facts across hops before producing an answer, separating internal computation from verbal explanation.
Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization
Mechanistically studies transformers that learn implicit reasoning after grokking, focusing on how internal algorithms support out-of-distribution generalization.
Distributional Reasoning in LLMs: Parallel Reasoning Processes in Multi-Hop Reasoning
Finds evidence for parallel internal processes during multi-hop reasoning rather than a single serial written-style chain.
Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process
Studies the internal computation behind grade-school mathematics, including when models learn robust procedures or rely on data shortcuts.
Uncovering Latent Chain of Thought Vectors in Language Models
Searches hidden activations for directions that encode intermediate reasoning progress and tests how those directions affect behavior.
Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluation
Uses the sequence of hidden states produced during inference to estimate whether an answer is correct, without extra training or labeled correctness examples.
Do LLMs Really Think Step-by-step In Implicit Reasoning?
Tests whether successful implicit reasoning reflects an internal step-by-step procedure and documents where that interpretation fails.
Implicit Language Models are RNNs: Balancing Parallelization and Expressivity
Connects implicit language models to recurrent computation and analyzes the trade-off between parallel training and expressive sequential depth.
Implicit Reasoning in Transformers is Reasoning through Shortcuts
Tests whether apparent implicit multi-step reasoning reflects compositional algorithms or learned shortcuts, emphasizing failure modes under distribution shift.
Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models
Encodes the answer in the language of the model's first response token, forcing a choice before any written explanation. Also tests whether successful answers could come from simple heuristics.
Do Language Models Use Their Depth Efficiently?
Skips layers and compares shallow and deep models to examine what extra layers compute. In the models studied, later layers mostly make smaller refinements rather than adding new reasoning steps.
Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMs
Presents evidence that transformer layers schedule intermediate subtasks internally, offering a layer-wise view of implicit reasoning.
Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning
Surveys latent chain-of-thought methods and organizes a broad, continuously updated bibliography spanning token-wise, structural, representational, and analytical work.
Language Models Can Learn Implicit Multi-Hop Reasoning, but Only If They Have Lots of Training Data
Studies the data requirements for learning implicit multi-hop reasoning and identifies sharp limitations in low-data regimes.
To CoT or To Loop? A Formal Comparison Between Chain-of-Thought and Looped Transformers
Formally compares explicit chain-of-thought computation with repeated internal transformations in looped transformer architectures.
Continuous Chain of Thought Enables Parallel Exploration and Reasoning
Analyzes and demonstrates how continuous intermediate states can explore several candidate reasoning paths in parallel.
Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer
Probes a depth-recurrent transformer to test what its repeated hidden-state computation represents and whether it behaves like a latent chain of thought.
A Survey on Latent Reasoning
Surveys latent-reasoning methods, evaluation, and applications and accompanies a maintained research bibliography.
LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinking
Probes soft-token reasoning and finds that tested models often follow the highest-probability component rather than exploring several paths at once. Introduces stochastic soft thinking, including Gumbel-Softmax sampling, to reduce this greedy behavior.
Implicit Reasoning in Large Language Models: A Comprehensive Survey
Organizes implicit-reasoning research across model mechanisms, training methods, evaluation, and applications.
Do Latent Tokens Think? A Causal and Adversarial Analysis of Chain-of-Continuous-Thought
Tests COCONUT with latent-state interventions and deliberately biased examples on MMLU and HotpotQA. Finds weak sensitivity to steering and dependence on dataset shortcuts in the evaluated models, challenging whether their intermediate states faithfully carry the reasoning.
Chain Of Thought Compression: A Theoretical Analysis
Analyzes how skipping intermediate steps weakens learning signals in a simplified parity-task model. Introduces NatBool-DAG to test logical dependencies and ALiCoT, which supervises learned token states with intermediate logical labels or explicit-CoT representations.
Do Latent-CoT Models Think Step-by-Step? A Mechanistic Study on Sequential Reasoning Tasks
Uses mechanistic interventions on sequential tasks to test whether latent chain-of-thought models actually execute stepwise internal procedures.
Capabilities and Fundamental Limits of Latent Chain-of-Thought
Characterizes tasks that latent chain-of-thought can express and identifies theoretical and empirical limits of the approach.
Transformers Provably Learn to Internalize Chain-of-Thought
Provides theoretical and empirical results on when transformers trained with a curriculum learn to internalize explicit reasoning steps.
Unlocking the Black Box of Latent Reasoning: An Interpretability-Guided Approach to Intervention
Uses probes, ablations, and targeted edits to study how continuous reasoning states affect answers. Builds interventions from learned semantic mappings and geometric constraints, improving the evaluated models by editing their latent states without updating base-model weights.
Observable Patterns Are Not Explanations: A Causal-Geometric Analysis of Latent Reasoning Models
Compares Coconut and CODI with controls lacking their proposed reasoning recurrence or curriculum. Finds that decodable patterns can also appear in controls, while interventions reveal varying causal use of latent states concentrated in low-dimensional directions.
Demystifying Hidden-State Recurrence: Switchable Latent Reasoning with On-Policy Reinforcement Learning
Trains switchable hidden-state recurrence with on-policy reinforcement learning and analyzes how the internal loop contributes to reasoning.
What Makes Effective Supervision in Latent Chain-of-Thought: An Information-Theoretic Analysis
Uses information-theoretic analysis to study which supervision signals make latent chain-of-thought training effective.
Think in Latent, Explain in Language: Self-Explainable Latent Reasoning
Jointly trains latent states for answer prediction and for decoding back into readable reasoning, using one model rather than a separate post-hoc explanation decoder.
Final Checkpoints Are Not Enough: Analyzing Latent Reasoning Faithfulness Along Training Trajectories
Tests how answers respond to changed inputs and altered hidden reasoning states throughout training. These responses can change even while accuracy remains high.
Training Continuous Chain of Thought Models: A Tale of Two Regimes
Introduces direct latent supervision through averaged trace embeddings and finds that both direct and indirect continuous-CoT training degrade sharply on tasks with substantially longer reasoning traces.
Does Out-of-Sight Equal Out-of-Mind in CoT Monitorability?
Compares monitoring from explicit traces, latent-state probes, and latent verbalizations, finding that monitorability depends strongly on task structure and level of internal access.
Think Shallow, Solve Deep: Controlling Recurrent Dynamics for Reliable Test-Time Depth
Studies when extra transformer loops improve an answer or make it worse. Gives a condition under which further loops cannot change the answer, and trains models toward this stable behavior.
Allocating Recurrent Compute in Looped Language Models
Studies which transformer components should recur and introduces MixerLoop, which repeats the sequence mixer while applying the dense feed-forward component once.
Detecting Hidden Chain-of-Thought in Large Language Models with Linguistic, Behavioral, and Mechanistic Indicators
Compares behavior under neutral, explicit-CoT, and no-CoT prompts using linguistic and internal-state measurements. Its score measures resemblance to CoT behavior, not direct evidence of a hidden reasoning trace; calibration controls expose important confounds.
Dissecting Hierarchical Reasoning Models: A Mechanistic Study
Uses state interventions, probes, and feature ablations to study how hierarchical recurrent models solve Sudoku, mazes, and ARC tasks. Compares hierarchical and single-state recurrence, and distinguishes information that can be read from a state from information that causally affects an answer.
A Score Is Not a Policy: Measuring the Value of Adaptive Revision
Tests when a hierarchical reasoner should revise the internal plan guiding its lower-level computation. Compares learned revision schedules with fixed schedules, separating whether a controller responds to its state from whether that response improves decisions.
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
Loops the middle half of a mixture-of-experts transformer twice while matching an unlooped baseline on computation, parameter count, and KV-cache size. Studies how the comparison changes with model scale and examines the attention patterns produced by revisiting layers.
Looped Transformers under the Jacobian Lens: Does the Global Workspace Survive Recurrence?
Tracks information that can be decoded from, and altered within, repeated transformer layers. Finds that Ouro reconstructs this information each loop, while Huginn carries it forward.
Towards Understanding Pause Token Fine-Tuning Dynamics: A Mode Retention Perspective
Places pause tokens at reasoning-step boundaries and masks their training loss. Studies how this affects learning new tasks while retaining earlier capabilities, including during reinforcement learning.
Fractal basins trap latent reasoning
Studies recurrent reasoning trajectories as dynamical systems on puzzles and mathematical logic. Links long solution times in the tested models to transient trapping near unstable states associated with nearly correct solutions.
Right Direction, Wrong Step: Geometric Analysis of Finite-Step Failure in Looped Transformers
Studies updates that initially increase support for a reference answer but overshoot when applied in full. Uses curvature along each update to explain failures and identify smaller useful steps.
How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents
Studies how growing a model during training, reusing layers, and modifying the connections between repeated blocks affect language-model scaling. Compares shared and unshared weights and operators that normalize or reintroduce earlier representations.
Storing Is Not Remembering: LSTM-UT and Bounded Gated Memory for Looped Transformers
Adds a small gated memory to a looped transformer and compares it with storing the full history of recurrent states. Uses a cellular-automaton task and language modeling to test whether memory remains useful as computation grows.
When Steering Fails in Latent Reasoning: A Latent-to-Language Transition Gap
Tests whether changing continuous reasoning states changes the model's later text output. Finds weaker steering effects than in explicit chain-of-thought, despite comparable changes to the hidden representations.
LOCKR: A Hidden-State Trajectory-Guided Planner for Detecting and Repairing Stable-but-Wrong Lock-In in Diffusion Language Models
Uses the history of diffusion hidden states to detect answers that become stable but remain wrong. A planner selects targeted repair branches and verifies continuations, testing whether these internal trajectories support better repair decisions than confidence or a single snapshot.
Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models
Studies when a recurrent language model settles on which context tokens to attend to. Reuses that selected working set in later loops while still recomputing attention weights and hidden-state updates.
Ceiling of a Task: When Can a Transformer Succeed Without Its Chain of Thought?
Analyzes tasks for which a fixed-depth transformer needs an informative chain of thought to exceed a task-specific accuracy limit. Combines circuit-based results with experiments that replace, erase, or shuffle reasoning tokens, including comparisons with filler tokens.
Beyond the Training Horizon: Mechanisms and Limits of Length Generalization in Looped Transformers
Compares how two looped-transformer training schemes solve multi-step tasks beyond their training length. Uses hidden-state interventions to identify how intermediate information moves through each model and where that process breaks down.
Why Deterministic PRM Guidance Underperforms in Discrete Diffusion Reasoning
Compares process-reward guidance with independent sampling and final-answer reranking at matched forward-pass budgets. In the tested diffusion models, early pruning removes promising candidates and intermediate-state scorers can be poor final judges; the generated solutions remain textual.
Reliable Parallel Decoding in Masked Diffusion Language Models
Studies when simultaneously committing several predicted tokens breaks their reasoning dependencies. RPD combines stability across final layers with confidence and uncertainty in preceding positions to defer unsupported predictions while retaining parallel text generation.
What Makes Recurrence Effective in Looped Language Models?
Compares loop layouts and ways of passing information between loops in language models. Examines when recurrence helps reasoning and when it helps less on knowledge-dependent tasks.
Scheduling Recursive Reasoning in Looped Transformers
Adjusts the size of each hidden-state update based on whether recent updates make consistent progress or fluctuate. Tests the resulting scheduler with and without additional training.
Looped Transformers as Optimizers
Treats each recurrent hidden-state update as an optimization step and derives loop designs from this view. Introduces OperLoop, combining adaptive step sizes and weight decay.
Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does
Uses graph-walk tasks to study how a looped transformer performs different computations with the same weights. Tests whether changing the state entering a loop can select its computation, using steering and activation-patching experiments.
What Limits Recursive Reasoning Models: Optimization, Architecture and Test-Time Scaling
Compares recursive reasoning models under a common training and evaluation setup. Isolates the effects of gradient propagation, batch size, state updates, and hierarchical structure on algorithmic tasks and generalization.
Scaling Laws for Looped Mixture of Experts
Fits scaling laws that jointly account for recurrent depth, expert sparsity, model size, and training data in looped mixture-of-experts models. Uses them to study how to allocate resources under compute and memory constraints.
Temporally-Resolved Token Attribution Reveals the Generation Dynamics of Diffusion Language Models
Extends integrated-gradient attribution across layers and denoising steps to examine which inputs influence a diffusion model's developing response. Includes multi-hop graph reasoning; the authors present attribution as a complement to, not a replacement for, causal intervention.
On the Chain-of-Thought Monitorability of Looped Language Models
Tests how recurrent depth affects the ability to monitor a language model through its written chain of thought. Finds task-dependent changes under stress tests, while matched comparisons do not show a general monitorability disadvantage for looped architectures.
Efficient Reasoning with Flow Language Models
Compares continuous flow and discrete diffusion models at matched sizes and denoising steps. A theoretical account and intermediate-state interventions examine whether retaining information about several candidates helps solve mazes and Sudoku before final discretization.
Safe at One Loop, Risky at Another: Aligning Safety Across Recurrent Depths in Looped Language Models
Evaluates harmful-query and jailbreak behavior at different recurrent depths of looped language models. Proposes depth-specific control, state sharing, and joint supervision to align safety behavior across depths.
S-Space: Exploring Spatial Workspace in Multimodal Language Models
Identifies spatial-coordinate directions in multimodal model activations and tests them through causal interventions. Separating perception from viewpoint transformation reveals cases where a model represents object locations but fails to perform the required spatial computation.
Learning to Extrapolate in Depth with Looped Transformers
Analyzes how many extra loops a model trained on graph connectivity can safely run. Derives a finite range where added depth solves harder graphs before further loops create false connections.
Knowledge transfer between simple discrete worlds
Evaluates a depth-recurrent model on Sudoku-like worlds with different rule combinations. Compares symbolic rule descriptions with solved examples when the test world contains new puzzles, new combinations, or unfamiliar rule instances.
Running Fewer Loops Removes Refusal Behaviour in Recurrent-Depth Language Models
Studies how reducing recurrent depth changes refusal behavior in two looped language-model families. Uses probes and interventions to distinguish recognizing a harmful request from refusing it, and tests safety training across randomly selected depths.
Resample the Answer: Where to Spend Test-Time Compute in Recursive Reasoning
Separately perturbs the answer state and the reasoning state of a recursive puzzle solver to study where extra search is useful. Uses the model's halting prediction to trigger restarts that resample the answer state.
Monitoring latent reasoning in looped language models
Inspects eight looped language models for state convergence, token decisions, readout quality, attention behavior, and sensitivity to injected noise. Finds important differences between models that receive the prompt once and those that reintroduce it at each loop.
Looped Reasoning Finishes Earlier Than You Think
Studies how oscillations can slow convergence in recurrent reasoning models. Proposes an architecture whose rotational state changes are separated from the representation used for reasoning, and tests convergence on puzzle-solving tasks.
Riding the Fixed-Point Curve: Image Classification in the Continuous Thought Machine
Analyzes the Continuous Thought Machine's repeated updates during image classification. Finds that its hidden states follow moving fixed points and that much of its attention schedule is shared across images.
Learning to Compute Is Not Learning to Retain or Reuse in Looped Transformers
Separates computing an answer, preserving it through additional loops, and keeping its hidden state usable for later steps. On state-machine and maze tasks, shows that training can improve one ability without securing the others.
Does Latent Chain-of-Thought Actually Reason? A Single, Unifying Taxonomy for Latent Reasoning
Uses chain deletion, content replacement, probes, and trajectory analysis across 21 model-task configurations. Separates what continuous states encode from whether the model uses their content, finding that dependence on the chain varies with task and training recipe.