AutoMMLab: Automatically Generating Deployable Models from Language
  Instructions for Computer Vision Tasks
Zekang Yang Wang Zeng Sheng Jin Chen Qian Ping Luo Wentao Liu
Automated machine learning (AutoML) is a collection of techniques designed to
automate the machine learning development process. While traditional AutoML
approaches have been successfully applied in several critical steps of model
development (e.g. hyperparameter optimization), there lacks a AutoML system
that automates the entire end-to-end model production workflow. To fill this
blank, we present AutoMMLab, a general-purpose LLM-empowered AutoML system that
follows user's language instructions to automate the whole model production
workflow for computer vision tasks. The proposed AutoMMLab system effectively
employs LLMs as the bridge to connect AutoML and OpenMMLab community,
empowering non-expert individuals to easily build task-specific models via a
user-friendly language interface. Specifically, we propose RU-LLaMA to
understand users' request and schedule the whole pipeline, and propose a novel
LLM-based hyperparameter optimizer called HPO-LLaMA to effectively search for
the optimal hyperparameters. Experiments show that our AutoMMLab system is
versatile and covers a wide range of mainstream tasks, including
classification, detection, segmentation and keypoint estimation. We further
develop a new benchmark, called LAMP, for studying key components in the
end-to-end prompt-based model training pipeline. Code, model, and data will be
released.