Dual-Path Temporal Map Optimization for Make-up Temporal Video Grounding
Jiaxiu Li Kun Li Jia Li Guoliang Chen Dan Guo Meng Wang
Make-up temporal video grounding (MTVG) aims to localize the target video
segment which is semantically related to a sentence describing a make-up
activity, given a long video. Compared with the general video grounding task,
MTVG focuses on meticulous actions and changes on the face. The make-up
instruction step, usually involving detailed differences in products and facial
areas, is more fine-grained than general activities (e.g, cooking activity and
furniture assembly). Thus, existing general approaches cannot locate the target
activity effectually. More specifically, existing proposal generation modules
are not yet fully developed in providing semantic cues for the more
fine-grained make-up semantic comprehension. To tackle this issue, we propose
an effective proposal-based framework named Dual-Path Temporal Map Optimization
Network (DPTMO) to capture fine-grained multimodal semantic details of make-up
activities. DPTMO extracts both query-agnostic and query-guided features to
construct two proposal sets and uses specific evaluation methods for the two
sets. Different from the commonly used single structure in previous methods,
our dual-path structure can mine more semantic information in make-up videos
and distinguish fine-grained actions well. These two candidate sets represent
the cross-modal makeup video-text similarity and multi-modal fusion
relationship, complementing each other. Each set corresponds to its respective
optimization perspective, and their joint prediction enhances the accuracy of
video timestamp prediction. Comprehensive experiments on the YouMakeup dataset
demonstrate our proposed dual structure excels in fine-grained semantic
comprehension.