Skip to content
conftrace_
2026 EMNLP EMNLP 2026

HuggingBrew: Brewing High-Quality Mid-Training Data from the HuggingFace Community Datasets

AbstractData curation for large language models (LLMs) has been extensively studied for both the pre-training and post-training stages, yet the intermediate mid-training phase—bridging general pretraining and task-specific fine-tuning—remains critically underexplored. Most existing mid-training data for state-of-the-art models remains proprietary. Meanwhile, the Hugging Face platform hosts a huge amount of community-contributed datasets, yet no systematic effort has been made to refine this rich ecosystem into a unified high-quality corpus. We present HuggingBrew, a large-scale mid-training corpus constructed through a principled, multi-step curation pipeline applied to the massive publicly accessible Hugging Face datasets. The pipeline operates at two complementary granularities: dataset-level operations act on each dataset independently, while corpus-level operations are applied jointly across all retained corpus. We construct three progressively curated variants—\textsfDraught, \textsfCraft, and \textsfReserve—containing 1.6T, 450B, and 40B tokens, respectively. Continual pretraining experiments under a 100B-token budget show that series yields substantial improvements over baselines across mathematical reasoning, general knowledge, and code generation, validating the untapped value of community-contributed data for mid-training.

More from EMNLP 2026

All EMNLP 2026 papers →