Rotate the Loop: Blockwise Conformality for Looped Transformers

Alexey Zemtsov, Albina Klepach, Vadim Lomshakov, Andrei Polubarov, Nikita Lyubaykin, Alexander Derevyagin, Vladislav Kurenkov, Alexander Nikulin

Splits the recurrent residual into groups and rotates each group separately, aiming to preserve information across repeated transformer loops. Evaluates state tracking and language modeling.

First public: September 30, 2026 · Added to this list: October 18, 2026