growing-transformers-modular-expansion

Build language models layer-by-layer on frozen embeddings, training new Transformer blocks sequentially while keeping lower layers frozen. Achieves 5% improvement over monolithic baselines on MMLU while fitting 740M trainable parameters per stage on single GPUs, enabling resource-efficient incremental scaling to 2.3B parameters.

Independent GitHub snapshot · Automated classification · Original documentation governs