Technical Report: Competition Solution For BetterMixture
Shuaijiang Zhao Xiaoquan Fang
In the era of flourishing large-scale models, the challenge of selecting and
optimizing datasets from the vast and complex sea of data, to enhance the
performance of large language models within the constraints of limited
computational resources, has become paramount. This paper details our solution
for the BetterMixture challenge, which focuses on the fine-tuning data mixing
for large language models. Our approach, which secured third place,
incorporates data deduplication, low-level and high-level quality filtering,
and diversity selection. The foundation of our solution is Ke-Data-Juicer, an
extension of Data-Juicer, demonstrating its robust capabilities in handling and
optimizing data for large language models.