VideoPro: A Visual Analytics Approach for Interactive Video Programming
Jianben He Xingbo Wang Kam Kwai Wong Xijie Huang Changjian Chen Zixin Chen Fengjie Wang Min Zhu Huamin Qu
Constructing supervised machine learning models for real-world video analysis
require substantial labeled data, which is costly to acquire due to scarce
domain expertise and laborious manual inspection. While data programming shows
promise in generating labeled data at scale with user-defined labeling
functions, the high dimensional and complex temporal information in videos
poses additional challenges for effectively composing and evaluating labeling
functions. In this paper, we propose VideoPro, a visual analytics approach to
support flexible and scalable video data programming for model steering with
reduced human effort. We first extract human-understandable events from videos
using computer vision techniques and treat them as atomic components of
labeling functions. We further propose a two-stage template mining algorithm
that characterizes the sequential patterns of these events to serve as labeling
function templates for efficient data labeling. The visual interface of
VideoPro facilitates multifaceted exploration, examination, and application of
the labeling templates, allowing for effective programming of video data at
scale. Moreover, users can monitor the impact of programming on model
performance and make informed adjustments during the iterative programming
process. We demonstrate the efficiency and effectiveness of our approach with
two case studies and expert interviews.