sparse-sparse-attention

Overcome the attention gap in sparse transformers by training with both full and sparse attention simultaneously, aligned through bidirectional losses that encourage naturally sparser distributions while maintaining learning capability, enabling efficient inference without capability degradation.

Independent GitHub snapshot · Automated classification · Original documentation governs