SnapCap: Efficient Snapshot Compressive Video Captioning
Jianqiao Sun Yudi Su Hao Zhang Ziheng Cheng Zequn Zeng Zhengjue Wang Bo Chen Xin Yuan
Video Captioning (VC) is a challenging multi-modal task since it requires
describing the scene in language by understanding various and complex videos.
For machines, the traditional VC follows the
"imaging-compression-decoding-and-then-captioning" pipeline, where compression
is pivot for storage and transmission. However, in such a pipeline, some
potential shortcomings are inevitable, i.e., information redundancy resulting
in low efficiency and information loss during the sampling process for
captioning. To address these problems, in this paper, we propose a novel VC
pipeline to generate captions directly from the compressed measurement, which
can be captured by a snapshot compressive sensing camera and we dub our model
SnapCap. To be more specific, benefiting from the signal simulation, we have
access to obtain abundant measurement-video-annotation data pairs for our
model. Besides, to better extract language-related visual representations from
the compressed measurement, we propose to distill the knowledge from videos via
a pre-trained CLIP with plentiful language-vision associations to guide the
learning of our SnapCap. To demonstrate the effectiveness of SnapCap, we
conduct experiments on two widely-used VC datasets. Both the qualitative and
quantitative results verify the superiority of our pipeline over conventional
VC pipelines. In particular, compared to the "caption-after-reconstruction"
methods, our SnapCap can run at least 3$\times$ faster, and achieve better
caption results.