Chain-of-Instructions: Compositional Instruction Tuning on Large
  Language Models
Shirley Anugrah Hayati Taehee Jung Tristan Bodding-Long Sudipta Kar Abhinav Sethy Joo-Kyung Kim Dongyeop Kang
Fine-tuning large language models (LLMs) with a collection of large and
diverse instructions has improved the model's generalization to different
tasks, even for unseen tasks. However, most existing instruction datasets
include only single instructions, and they struggle to follow complex
instructions composed of multiple subtasks (Wang et al., 2023a). In this work,
we propose a novel concept of compositional instructions called
chain-of-instructions (CoI), where the output of one instruction becomes an
input for the next like a chain. Unlike the conventional practice of solving
single instruction tasks, our proposed method encourages a model to solve each
subtask step by step until the final answer is reached. CoI-tuning (i.e.,
fine-tuning with CoI instructions) improves the model's ability to handle
instructions composed of multiple subtasks. CoI-tuned models also outperformed
baseline models on multilingual summarization, demonstrating the
generalizability of CoI models on unseen composite downstream tasks.