The Visual Language of Fabrics
Valentin Deschaintre Julia Guerrero-Viu Diego Gutierrez Tamy Boubekeur Belen Masia
We introduce text2fabric, a novel dataset that links free-text descriptions
to various fabric materials. The dataset comprises 15,000 natural language
descriptions associated to 3,000 corresponding images of fabric materials.
Traditionally, material descriptions come in the form of tags/keywords, which
limits their expressivity, induces pre-existing knowledge of the appropriate
vocabulary, and ultimately leads to a chopped description system. Therefore, we
study the use of free-text as a more appropriate way to describe material
appearance, taking the use case of fabrics as a common item that non-experts
may often deal with. Based on the analysis of the dataset, we identify a
compact lexicon, set of attributes and key structure that emerge from the
descriptions. This allows us to accurately understand how people describe
fabrics and draw directions for generalization to other types of materials. We
also show that our dataset enables specializing large vision-language models
such as CLIP, creating a meaningful latent space for fabric appearance, and
significantly improving applications such as fine-grained material retrieval
and automatic captioning.