growing-transformers-modular-expansion

【效率与协作】Build language models layer-by-layer on frozen embeddings, training new Transformer blocks sequentially while keeping lower layers frozen. Achieves 5% improvement over monolithic baselines on MMLU while fitting 740M trainable parameters per stage on single GPUs, enabling resource-efficient incremental scaling to 2.3B parameters.

独立 GitHub 快照 · 自动分类 · 以原作者文档为准