UpLoop: Training Looped Transformers via Upcycling at No Extra Pretraining Compute

Cheng Liang, Tengfei Zhang, Ya Zhang, Yanfeng Wang, Zhouhan Lin

Pretrains a shallow transformer, grows it by copying layers and optimizer states, then adds shared-weight repetitions of its middle block. Uses the compute saved early in training to match a dense baseline in parameters, training tokens, and total training FLOPs.

First public: October 30, 2026 · Added to this list: November 2, 2026