OCR is All you need: Importing Multi-Modality into Image-based Defect
  Detection System
Chih-Chung Hsu Chia-Ming Lee Chun-Hung Sun Kuang-Ming Wu
Automatic optical inspection (AOI) plays a pivotal role in the manufacturing
process, predominantly leveraging high-resolution imaging instruments for
scanning purposes. It detects anomalies by analyzing image textures or
patterns, making it an essential tool in industrial manufacturing and quality
control. Despite its importance, the deployment of models for AOI often faces
challenges. These include limited sample sizes, which hinder effective feature
learning, variations among source domains, and sensitivities to changes in
lighting and camera positions during imaging. These factors collectively
compromise the accuracy of model predictions. Traditional AOI often fails to
capitalize on the rich mechanism-parameter information from machines or inside
images, including statistical parameters, which typically benefit AOI
classification. To address this, we introduce an external modality-guided data
mining framework, primarily rooted in optical character recognition (OCR), to
extract statistical features from images as a second modality to enhance
performance, termed OANet (Ocr-Aoi-Net). A key aspect of our approach is the
alignment of external modality features, extracted using a single
modality-aware model, with image features encoded by a convolutional neural
network. This synergy enables a more refined fusion of semantic representations
from different modalities. We further introduce feature refinement and a gating
function in our OANet to optimize the combination of these features, enhancing
inference and decision-making capabilities. Experimental outcomes show that our
methodology considerably boosts the recall rate of the defect detection model
and maintains high robustness even in challenging scenarios.