Balancing Specialized and General Skills in LLMs: The Impact of Modern
  Tuning and Data Strategy
Zheng Zhang Chen Zheng Da Tang Ke Sun Yukun Ma Yingtong Bu Xun Zhou Liang Zhao
This paper introduces a multifaceted methodology for fine-tuning and
evaluating large language models (LLMs) for specialized monetization tasks. The
goal is to balance general language proficiency with domain-specific skills.
The methodology has three main components: 1) Carefully blending in-domain and
general-purpose data during fine-tuning to achieve an optimal balance between
general and specialized capabilities; 2) Designing a comprehensive evaluation
framework with 45 questions tailored to assess performance on functionally
relevant dimensions like reliability, consistency, and business impact; 3)
Analyzing how model size and continual training influence metrics to guide
efficient resource allocation during fine-tuning. The paper details the design,
data collection, analytical techniques, and results validating the proposed
frameworks. It aims to provide businesses and researchers with actionable
insights on effectively adapting LLMs for specialized contexts. We also intend
to make public the comprehensive evaluation framework, which includes the 45
tailored questions and their respective scoring guidelines, to foster
transparency and collaboration in adapting LLMs for specialized tasks.