D-Cubed: Latent Diffusion Trajectory Optimisation for Dexterous
  Deformable Manipulation
Jun Yamada Shaohong Zhong Jack Collins Ingmar Posner
Mastering dexterous robotic manipulation of deformable objects is vital for
overcoming the limitations of parallel grippers in real-world applications.
Current trajectory optimisation approaches often struggle to solve such tasks
due to the large search space and the limited task information available from a
cost function. In this work, we propose D-Cubed, a novel trajectory
optimisation method using a latent diffusion model (LDM) trained from a
task-agnostic play dataset to solve dexterous deformable object manipulation
tasks. D-Cubed learns a skill-latent space that encodes short-horizon actions
in the play dataset using a VAE and trains a LDM to compose the skill latents
into a skill trajectory, representing a long-horizon action trajectory in the
dataset. To optimise a trajectory for a target task, we introduce a novel
gradient-free guided sampling method that employs the Cross-Entropy method
within the reverse diffusion process. In particular, D-Cubed samples a small
number of noisy skill trajectories using the LDM for exploration and evaluates
the trajectories in simulation. Then, D-Cubed selects the trajectory with the
lowest cost for the subsequent reverse process. This effectively explores
promising solution areas and optimises the sampled trajectories towards a
target task throughout the reverse diffusion process. Through empirical
evaluation on a public benchmark of dexterous deformable object manipulation
tasks, we demonstrate that D-Cubed outperforms traditional trajectory
optimisation and competitive baseline approaches by a significant margin. We
further demonstrate that trajectories found by D-Cubed readily transfer to a
real-world LEAP hand on a folding task.