Token-Efficient Leverage Learning in Large Language Models
Yuanhao Zeng Min Wang Yihang Wang Yingxia Shao
Large Language Models (LLMs) have excelled in various tasks but perform
better in high-resource scenarios, which presents challenges in low-resource
scenarios. Data scarcity and the inherent difficulty of adapting LLMs to
specific tasks compound the challenge. To address the twin hurdles, we
introduce \textbf{Leverage Learning}. We present a streamlined implement of
this methodology called Token-Efficient Leverage Learning (TELL). TELL
showcases the potential of Leverage Learning, demonstrating effectiveness
across various LLMs and low-resource tasks, ranging from $10^4$ to $10^6$
tokens. It reduces task data requirements by up to nearly an order of magnitude
compared to conventional Supervised Fine-Tuning (SFT) while delivering
competitive performance. With the same amount of task data, TELL leads in
improving task performance compared to SFT. We discuss the mechanism of
Leverage Learning, suggesting it aligns with quantization hypothesis and
explore its promising potential through empirical testing.