pretraining-data-refinement

Improve pretraining efficiency by refining noisy data through expert-guided programs: learn to generate deletion operations that clean documents, achieving 2.6-7.2% performance gains with fewer training tokens.

Independent GitHub snapshot · Automated classification · Original documentation governs