Learning to Find Missing Video Frames with Synthetic Data Augmentation:
  A General Framework and Application in Generating Thermal Images Using RGB
  Cameras
Mathias Viborg Andersen Ross Greer Andreas Møgelmose Mohan Trivedi
Advanced Driver Assistance Systems (ADAS) in intelligent vehicles rely on
accurate driver perception within the vehicle cabin, often leveraging a
combination of sensing modalities. However, these modalities operate at varying
rates, posing challenges for real-time, comprehensive driver state monitoring.
This paper addresses the issue of missing data due to sensor frame rate
mismatches, introducing a generative model approach to create synthetic yet
realistic thermal imagery. We propose using conditional generative adversarial
networks (cGANs), specifically comparing the pix2pix and CycleGAN
architectures. Experimental results demonstrate that pix2pix outperforms
CycleGAN, and utilizing multi-view input styles, especially stacked views,
enhances the accuracy of thermal image generation. Moreover, the study
evaluates the model's generalizability across different subjects, revealing the
importance of individualized training for optimal performance. The findings
suggest the potential of generative models in addressing missing frames,
advancing driver state monitoring for intelligent vehicles, and underscoring
the need for continued research in model generalization and customization.