sonicmoe-training-optimization

【设计与创意】Optimize MoE training through memory-efficient backward pass, IO-aware kernel design overlapping memory operations, and token rounding routing. Avoid caching large-scale activations, fuse operations with GEMM, implement ping-pong scheduling. Achieve 1.86× compute throughput on Hopper GPUs and 16% higher TFLOPS in sparse configurations.

独立 GitHub 快照 · 自动分类 · 以原作者文档为准