Loop-μP: Residual Scaling and Hyperparameter Transfer in Looped Transformers
Scales each residual update inversely with the number of shared-weight loops, while keeping the learning rate independent of that loop count. Derives the scaling rule and tests whether hyperparameters transfer across loop counts and model depths during language-model pretraining.
First public: · Added to this list: November 3, 2026
Earlier title: On the Residual Scaling of Looped Transformers: Stability and Transferability.
Explore this approach
Recurrent reasoning — Repeat parts of the model or keep intermediate information in memory while working through a problem. Some methods also learn how many passes to run.
Analysis and evaluation — Measure how well these methods work, investigate what their internal computation does, and identify where they fail. This includes experiments, studies of model internals, and mathematical results.