simple-gpt-normalization-strategy

【效率与协作】Place RMSNorm immediately after every linear layer to stabilize activation scales at O(sqrt(d)) and reduce Hessian spectral norm. Enables 3-10x larger learning rates and faster convergence without architectural changes; improves loss by 0.08 on 7B models.

独立 GitHub 快照 · 自动分类 · 以原作者文档为准