Self-Instruct: Aligning Language Models with Self-Generated Instructions

Self-Instruct is a framework for improving the instruction-following capabilities of pretrained language models by bootstrapping off their own generations, addressing the problem that instruction-tuned models depend heavily on human-written instruction data that is limited in quantity, diversity, and creativity. The pipeline uses a language model to generate instructions along with corresponding input and output samples, then filters out invalid or overly similar examples before using the remaining data to finetune the original model. This provides an almost annotation-free method for aligning pretrained language models with instructions.

Applying the method to the vanilla GPT3, the authors demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, putting it on par with InstructGPT-001, which was trained with private user data and human annotations. On a curated set of expert-written instructions for novel tasks, human evaluation shows Self-Instruct tuning outperforms existing public instruction datasets by a large margin, leaving only a 5% absolute gap behind InstructGPT-001. The authors release their large synthetic dataset to facilitate future work on instruction tuning.