Listen, Chat, and Edit: Text-Guided Soundscape Modification for Enhanced
  Auditory Experience
Xilin Jiang Cong Han Yinghao Aaron Li Nima Mesgarani
In daily life, we encounter a variety of sounds, both desirable and
undesirable, with limited control over their presence and volume. Our work
introduces "Listen, Chat, and Edit" (LCE), a novel multimodal sound mixture
editor that modifies each sound source in a mixture based on user-provided text
instructions. LCE distinguishes itself with a user-friendly chat interface and
its unique ability to edit multiple sound sources simultaneously within a
mixture, without needing to separate them. Users input open-vocabulary text
prompts, which are interpreted by a large language model to create a semantic
filter for editing the sound mixture. The system then decomposes the mixture
into its components, applies the semantic filter, and reassembles it into the
desired output. We developed a 160-hour dataset with over 100k mixtures,
including speech and various audio sources, along with text prompts for diverse
editing tasks like extraction, removal, and volume control. Our experiments
demonstrate significant improvements in signal quality across all editing tasks
and robust performance in zero-shot scenarios with varying numbers and types of
sound sources.