Multi-dimensional data refining strategy for effective fine-tuning LLMs
Thanh Nguyen Ngoc Quang Nhat Tran Arthur Tang Bao Nguyen Thuy Nguyen Thanh Pham
Data is a cornerstone for fine-tuning large language models, yet acquiring
suitable data remains challenging. Challenges encompassed data scarcity,
linguistic diversity, and domain-specific content. This paper presents lessons
learned while crawling and refining data tailored for fine-tuning Vietnamese
language models. Crafting such a dataset, while accounting for linguistic
intricacies and striking a balance between inclusivity and accuracy, demands
meticulous planning. Our paper presents a multidimensional strategy including
leveraging existing datasets in the English language and developing customized
data-crawling scripts with the assistance of generative AI tools. A fine-tuned
LLM model for the Vietnamese language, which was produced using resultant
datasets, demonstrated good performance while generating Vietnamese news
articles from prompts. The study offers practical solutions and guidance for
future fine-tuning models in languages like Vietnamese.