Conditional Generative Modeling for Images, 3D Animations, and Video
Vikram Voleti
This dissertation attempts to drive innovation in the field of generative
modeling for computer vision, by exploring novel formulations of conditional
generative models, and innovative applications in images, 3D animations, and
video. Our research focuses on architectures that offer reversible
transformations of noise and visual data, and the application of
encoder-decoder architectures for generative tasks and 3D content manipulation.
In all instances, we incorporate conditional information to enhance the
synthesis of visual data, improving the efficiency of the generation process as
well as the generated content.
  We introduce the use of Neural ODEs to model video dynamics using an
encoder-decoder architecture, demonstrating their ability to predict future
video frames despite being trained solely to reconstruct current frames. Next,
we propose a conditional variant of continuous normalizing flows that enables
higher-resolution image generation based on lower-resolution input, achieving
comparable image quality while reducing parameters and training time. Our next
contribution presents a pipeline that takes human images as input,
automatically aligns a user-specified 3D character with the pose of the human,
and facilitates pose editing based on partial inputs. Next, we derive the
relevant mathematical details for denoising diffusion models that use
non-isotropic Gaussian processes, and show comparable generation quality.
Finally, we devise a novel denoising diffusion framework capable of solving all
three video tasks of prediction, generation, and interpolation. We perform
ablation studies, and show SOTA results on multiple datasets.
  Our contributions are published articles at peer-reviewed venues. Overall,
our research aims to make a meaningful contribution to the pursuit of more
efficient and flexible generative models, with the potential to shape the
future of computer vision.