Bridging Classical and Quantum Machine Learning: Knowledge Transfer From
  Classical to Quantum Neural Networks Using Knowledge Distillation
Mohammad Junayed Hasan M. R. C. Mahdy
Very recently, studies have shown that quantum neural networks surpass
classical neural networks in tasks like image classification when a similar
number of learnable parameters are used. However, the development and
optimization of quantum models are currently hindered by issues such as qubit
instability and limited qubit availability, leading to error-prone systems with
weak performance. In contrast, classical models can exhibit high-performance
owing to substantial resource availability. As a result, more studies have been
focusing on hybrid classical-quantum integration. A line of research
particularly focuses on transfer learning through classical-quantum integration
or quantum-quantum approaches. Unlike previous studies, this paper introduces a
new method to transfer knowledge from classical to quantum neural networks
using knowledge distillation, effectively bridging the gap between classical
machine learning and emergent quantum computing techniques. We adapt classical
convolutional neural network (CNN) architectures like LeNet and AlexNet to
serve as teacher networks, facilitating the training of student quantum models
by sending supervisory signals during backpropagation through KL-divergence.
The approach yields significant performance improvements for the quantum models
by solely depending on classical CNNs, with quantum models achieving an average
accuracy improvement of 0.80% on the MNIST dataset and 5.40% on the more
complex Fashion MNIST dataset. Applying this technique eliminates the
cumbersome training of huge quantum models for transfer learning in
resource-constrained settings and enables re-using existing pre-trained
classical models to improve performance.Thus, this study paves the way for
future research in quantum machine learning (QML) by positioning knowledge
distillation as a core technique for advancing QML applications.