Progressive Alignment with VLM-LLM Feature to Augment Defect
  Classification for the ASE Dataset
Chih-Chung Hsu Chia-Ming Lee Chun-Hung Sun Kuang-Ming Wu
Traditional defect classification approaches are facing with two barriers.
(1) Insufficient training data and unstable data quality. Collecting sufficient
defective sample is expensive and time-costing, consequently leading to dataset
variance. It introduces the difficulty on recognition and learning. (2)
Over-dependence on visual modality. When the image pattern and texture is
monotonic for all defect classes in a given dataset, the performance of
conventional AOI system cannot be guaranteed. In scenarios where image quality
is compromised due to mechanical failures or when defect information is
inherently difficult to discern, the performance of deep models cannot be
guaranteed. A main question is, "how to solve those two problems when they
occur at the same time?" The feasible strategy is to explore another feature
within dataset and combine an eminent vision-language model (VLM) and
Large-Language model (LLM) with their astonishing zero-shot capability. In this
work, we propose the special ASE dataset, including rich data description
recorded on image, for defect classification, but the defect feature is uneasy
to learn directly. Secondly, We present the prompting for VLM-LLM against
defect classification with the proposed ASE dataset to activate extra-modality
feature from images to enhance performance. Then, We design the novel
progressive feature alignment (PFA) block to refine image-text feature to
alleviate the difficulty of alignment under few-shot scenario. Finally, the
proposed Cross-modality attention fusion (CMAF) module can effectively fuse
different modality feature. Experiment results have demonstrated our method's
effectiveness over several defect classification methods for the ASE dataset.