Engagement Measurement Based on Facial Landmarks and Spatial-Temporal
  Graph Convolutional Networks
Ali Abedi Shehroz S. Khan
Engagement in virtual learning is crucial for a variety of factors including
learner satisfaction, performance, and compliance with learning programs, but
measuring it is a challenging task. There is therefore considerable interest in
utilizing artificial intelligence and affective computing to measure engagement
in natural settings as well as on a large scale. This paper introduces a novel,
privacy-preserving method for engagement measurement from videos. It uses
facial landmarks, which carry no personally identifiable information, extracted
from videos via the MediaPipe deep learning solution. The extracted facial
landmarks are fed to a Spatial-Temporal Graph Convolutional Network (ST-GCN) to
output the engagement level of the learner in the video. To integrate the
ordinal nature of the engagement variable into the training process, ST-GCNs
undergo training in a novel ordinal learning framework based on transfer
learning. Experimental results on two video student engagement measurement
datasets show the superiority of the proposed method compared to previous
methods with improved state-of-the-art on the EngageNet dataset with a %3.1
improvement in four-class engagement level classification accuracy and on the
Online Student Engagement dataset with a %1.5 improvement in binary engagement
classification accuracy. The relatively lightweight ST-GCN and its integration
with the real-time MediaPipe deep learning solution make the proposed approach
capable of being deployed on virtual learning platforms and measuring
engagement in real time.