DragD3D: Vertex-based Editing for Realistic Mesh Deformations using 2D
  Diffusion Priors
Tianhao Xie Eugene Belilovsky Sudhir Mudur Tiberiu Popa
Direct mesh editing and deformation are key components in the geometric
modeling and animation pipeline. Direct mesh editing methods are typically
framed as optimization problems combining user-specified vertex constraints
with a regularizer that determines the position of the rest of the vertices.
The choice of the regularizer is key to the realism and authenticity of the
final result. Physics and geometry-based regularizers are not aware of the
global context and semantics of the object, and the more recent deep learning
priors are limited to a specific class of 3D object deformations. In this work,
our main contribution is a local mesh editing method called DragD3D for global
context-aware realistic deformation through direct manipulation of a few
vertices. DragD3D is not restricted to any class of objects. It achieves this
by combining the classic geometric ARAP (as rigid as possible) regularizer with
2D priors obtained from a large-scale diffusion model. Specifically, we render
the objects from multiple viewpoints through a differentiable renderer and use
the recently introduced DDS loss which scores the faithfulness of the rendered
image to one from a diffusion model. DragD3D combines the approximate gradients
of the DDS with gradients from the ARAP loss to modify the mesh vertices via
neural Jacobian field, while also satisfying vertex constraints. We show that
our deformations are realistic and aware of the global context of the objects,
and provide better results than just using geometric regularizers.