Materialistic: Selecting Similar Materials in Images
Prafull Sharma Julien Philip Michaël Gharbi William T. Freeman Fredo Durand Valentin Deschaintre
Separating an image into meaningful underlying components is a crucial first
step for both editing and understanding images. We present a method capable of
selecting the regions of a photograph exhibiting the same material as an
artist-chosen area. Our proposed approach is robust to shading, specular
highlights, and cast shadows, enabling selection in real images. As we do not
rely on semantic segmentation (different woods or metal should not be selected
together), we formulate the problem as a similarity-based grouping problem
based on a user-provided image location. In particular, we propose to leverage
the unsupervised DINO features coupled with a proposed Cross-Similarity module
and an MLP head to extract material similarities in an image. We train our
model on a new synthetic image dataset, that we release. We show that our
method generalizes well to real-world images. We carefully analyze our model's
behavior on varying material properties and lighting. Additionally, we evaluate
it against a hand-annotated benchmark of 50 real photographs. We further
demonstrate our model on a set of applications, including material editing,
in-video selection, and retrieval of object photographs with similar materials.