Customizing Visual-Language Foundation Models for Multi-modal Anomaly
  Detection and Reasoning
Xiaohao Xu Yunkang Cao Yongqi Chen Weiming Shen Xiaonan Huang
Anomaly detection is vital in various industrial scenarios, including the
identification of unusual patterns in production lines and the detection of
manufacturing defects for quality control. Existing techniques tend to be
specialized in individual scenarios and lack generalization capacities. In this
study, we aim to develop a generic anomaly detection model applicable across
multiple scenarios. To achieve this, we customize generic visual-language
foundation models that possess extensive knowledge and robust reasoning
abilities into anomaly detectors and reasoners. Specifically, we introduce a
multi-modal prompting strategy that incorporates domain knowledge from experts
as conditions to guide the models. Our approach considers multi-modal prompt
types, including task descriptions, class context, normality rules, and
reference images. In addition, we unify the input representation of
multi-modality into a 2D image format, enabling multi-modal anomaly detection
and reasoning. Our preliminary studies demonstrate that combining visual and
language prompts as conditions for customizing the models enhances anomaly
detection performance. The customized models showcase the ability to detect
anomalies across different data modalities such as images and point clouds.
Qualitative case studies further highlight the anomaly detection and reasoning
capabilities, particularly for multi-object scenes and temporal data. Our code
is available at https://github.com/Xiaohao-Xu/Customizable-VLM.