Learning Through Guidance: Knowledge Distillation for Endoscopic Image
  Classification
Harshala Gammulle Yubo Chen Sridha Sridharan Travis Klein Clinton Fookes
Endoscopy plays a major role in identifying any underlying abnormalities
within the gastrointestinal (GI) tract. There are multiple GI tract diseases
that are life-threatening, such as precancerous lesions and other intestinal
cancers. In the usual process, a diagnosis is made by a medical expert which
can be prone to human errors and the accuracy of the test is also entirely
dependent on the expert's level of experience. Deep learning, specifically
Convolution Neural Networks (CNNs) which are designed to perform automatic
feature learning without any prior feature engineering, has recently reported
great benefits for GI endoscopy image analysis. Previous research has developed
models that focus only on improving performance, as such, the majority of
introduced models contain complex deep network architectures with a large
number of parameters that require longer training times. However, there is a
lack of focus on developing lightweight models which can run in low-resource
environments, which are typically encountered in medical clinics. We
investigate three KD-based learning frameworks, response-based, feature-based,
and relation-based mechanisms, and introduce a novel multi-head attention-based
feature fusion mechanism to support relation-based learning. Compared to the
existing relation-based methods that follow simplistic aggregation techniques
of multi-teacher response/feature-based knowledge, we adopt the multi-head
attention technique to provide flexibility towards localising and transferring
important details from each teacher to better guide the student. We perform
extensive evaluations on two widely used public datasets, KVASIR-V2 and
Hyper-KVASIR, and our experimental results signify the merits of our proposed
relation-based framework in achieving an improved lightweight model (only 51.8k
trainable parameters) that can run in a resource-limited environment.