An Initial Exploration: Learning to Generate Realistic Audio for Silent
  Video
Matthew Martel Jackson Wagner
Generating realistic audio effects for movies and other media is a
challenging task that is accomplished today primarily through physical
techniques known as Foley art. Foley artists create sounds with common objects
(e.g., boxing gloves, broken glass) in time with video as it is playing to
generate captivating audio tracks. In this work, we aim to develop a
deep-learning based framework that does much the same - observes video in it's
natural sequence and generates realistic audio to accompany it. Notably, we
have reason to believe this is achievable due to advancements in realistic
audio generation techniques conditioned on other inputs (e.g., Wavenet
conditioned on text). We explore several different model architectures to
accomplish this task that process both previously-generated audio and video
context. These include deep-fusion CNN, dilated Wavenet CNN with visual
context, and transformer-based architectures. We find that the
transformer-based architecture yields the most promising results, matching
low-frequencies to visual patterns effectively, but failing to generate more
nuanced waveforms.