VIoTGPT: Learning to Schedule Vision Tools towards Intelligent Video
  Internet of Things
Yaoyao Zhong Mengshi Qi Rui Wang Yuhan Qiu Yang Zhang Huadong Ma
Video Internet of Things (VIoT) has shown full potential in collecting an
unprecedented volume of video data. Learning to schedule perceiving models and
analyzing the collected videos intelligently will be potential sparks for VIoT.
In this paper, to address the challenges posed by the fine-grained and
interrelated vision tool usage of VIoT, we build VIoTGPT, the framework based
on LLMs to correctly interact with humans, query knowledge videos, and invoke
vision models to accomplish complicated tasks. To support VIoTGPT and related
future works, we meticulously crafted the training dataset and established
benchmarks involving 11 representative vision models across three categories
based on semi-automatic annotations. To guide LLM to act as the intelligent
agent towards intelligent VIoT, we resort to ReAct instruction tuning based on
the collected VIoT dataset to learn the tool capability. Quantitative and
qualitative experimental results and analyses demonstrate the effectiveness of
VIoTGPT.