TinyVQA: Compact Multimodal Deep Neural Network for Visual Question
  Answering on Resource-Constrained Devices
Hasib-Al Rashid Argho Sarkar Aryya Gangopadhyay Maryam Rahnemoonfar Tinoosh Mohsenin
Traditional machine learning models often require powerful hardware, making
them unsuitable for deployment on resource-limited devices. Tiny Machine
Learning (tinyML) has emerged as a promising approach for running machine
learning models on these devices, but integrating multiple data modalities into
tinyML models still remains a challenge due to increased complexity, latency,
and power consumption. This paper proposes TinyVQA, a novel multimodal deep
neural network for visual question answering tasks that can be deployed on
resource-constrained tinyML hardware. TinyVQA leverages a supervised
attention-based model to learn how to answer questions about images using both
vision and language modalities. Distilled knowledge from the supervised
attention-based VQA model trains the memory aware compact TinyVQA model and low
bit-width quantization technique is employed to further compress the model for
deployment on tinyML devices. The TinyVQA model was evaluated on the FloodNet
dataset, which is used for post-disaster damage assessment. The compact model
achieved an accuracy of 79.5%, demonstrating the effectiveness of TinyVQA for
real-world applications. Additionally, the model was deployed on a Crazyflie
2.0 drone, equipped with an AI deck and GAP8 microprocessor. The TinyVQA model
achieved low latencies of 56 ms and consumes 693 mW power while deployed on the
tiny drone, showcasing its suitability for resource-constrained embedded
systems.