Collaborative Neural Painting
Nicola Dall'Asen Willi Menapace Elia Peruzzo Enver Sangineto Yiming Wang Elisa Ricci
The process of painting fosters creativity and rational planning. However,
existing generative AI mostly focuses on producing visually pleasant artworks,
without emphasizing the painting process. We introduce a novel task,
Collaborative Neural Painting (CNP), to facilitate collaborative art painting
generation between humans and machines. Given any number of user-input
brushstrokes as the context or just the desired object class, CNP should
produce a sequence of strokes supporting the completion of a coherent painting.
Importantly, the process can be gradual and iterative, so allowing users'
modifications at any phase until the completion. Moreover, we propose to solve
this task using a painting representation based on a sequence of parametrized
strokes, which makes it easy both editing and composition operations. These
parametrized strokes are processed by a Transformer-based architecture with a
novel attention mechanism to model the relationship between the input strokes
and the strokes to complete. We also propose a new masking scheme to reflect
the interactive nature of CNP and adopt diffusion models as the basic learning
process for its effectiveness and diversity in the generative field. Finally,
to develop and validate methods on the novel task, we introduce a new dataset
of painted objects and an evaluation protocol to benchmark CNP both
quantitatively and qualitatively. We demonstrate the effectiveness of our
approach and the potential of the CNP task as a promising avenue for future
research.