Generating Visual Scenes from Touch
Fengyu Yang Jiacheng Zhang Andrew Owens
An emerging line of work has sought to generate plausible imagery from touch.
Existing approaches, however, tackle only narrow aspects of the visuo-tactile
synthesis problem, and lag significantly behind the quality of cross-modal
synthesis methods in other domains. We draw on recent advances in latent
diffusion to create a model for synthesizing images from tactile signals (and
vice versa) and apply it to a number of visuo-tactile synthesis tasks. Using
this model, we significantly outperform prior work on the tactile-driven
stylization problem, i.e., manipulating an image to match a touch signal, and
we are the first to successfully generate images from touch without additional
sources of information about the scene. We also successfully use our model to
address two novel synthesis problems: generating images that do not contain the
touch sensor or the hand holding it, and estimating an image's shading from its
reflectance and touch.