Tensor Factorization for Leveraging Cross-Modal Knowledge in
  Data-Constrained Infrared Object Detection
Manish Sharma Moitreya Chatterjee Kuan-Chuan Peng Suhas Lohit Michael Jones
The primary bottleneck towards obtaining good recognition performance in IR
images is the lack of sufficient labeled training data, owing to the cost of
acquiring such data. Realizing that object detection methods for the RGB
modality are quite robust (at least for some commonplace classes, like person,
car, etc.), thanks to the giant training sets that exist, in this work we seek
to leverage cues from the RGB modality to scale object detectors to the IR
modality, while preserving model performance in the RGB modality. At the core
of our method, is a novel tensor decomposition method called TensorFact which
splits the convolution kernels of a layer of a Convolutional Neural Network
(CNN) into low-rank factor matrices, with fewer parameters than the original
CNN. We first pretrain these factor matrices on the RGB modality, for which
plenty of training data are assumed to exist and then augment only a few
trainable parameters for training on the IR modality to avoid over-fitting,
while encouraging them to capture complementary cues from those trained only on
the RGB modality. We validate our approach empirically by first assessing how
well our TensorFact decomposed network performs at the task of detecting
objects in RGB images vis-a-vis the original network and then look at how well
it adapts to IR images of the FLIR ADAS v1 dataset. For the latter, we train
models under scenarios that pose challenges stemming from data paucity. From
the experiments, we observe that: (i) TensorFact shows performance gains on RGB
images; (ii) further, this pre-trained model, when fine-tuned, outperforms a
standard state-of-the-art object detector on the FLIR ADAS v1 dataset by about
4% in terms of mAP 50 score.