R2-Diff: Denoising by diffusion as a refinement of retrieved motion for
  image-based motion prediction
Takeru Oba Norimichi Ukita
Image-based motion prediction is one of the essential techniques for robot
manipulation. Among the various prediction models, we focus on diffusion models
because they have achieved state-of-the-art performance in various
applications. In image-based motion prediction, diffusion models stochastically
predict contextually appropriate motion by gradually denoising random Gaussian
noise based on the image context. While diffusion models are able to predict
various motions by changing the random noise, they sometimes fail to predict a
contextually appropriate motion based on the image because the random noise is
sampled independently of the image context. To solve this problem, we propose
R2-Diff. In R2-Diff, a motion retrieved from a dataset based on image
similarity is fed into a diffusion model instead of random noise. Then, the
retrieved motion is refined through the denoising process of the diffusion
model. Since the retrieved motion is almost appropriate to the context, it
becomes easier to predict contextually appropriate motion. However, traditional
diffusion models are not optimized to refine the retrieved motion. Therefore,
we propose the method of tuning the hyperparameters based on the distance of
the nearest neighbor motion among the dataset to optimize the diffusion model
for refinement. Furthermore, we propose an image-based retrieval method to
retrieve the nearest neighbor motion in inference. Our proposed retrieval
efficiently computes the similarity based on the image features along the
motion trajectory. We demonstrate that R2-Diff accurately predicts appropriate
motions and achieves high task success rates compared to recent
state-of-the-art models in robot manipulation.