MTGA: Multi-view Temporal Granularity aligned Aggregation for
  Event-based Lip-reading
Wenhao Zhang Jun Wang Yong Luo Lei Yu Wei Yu Zheng He
Lip-reading is to utilize the visual information of the speaker's lip
movements to recognize words and sentences. Existing event-based lip-reading
solutions integrate different frame rate branches to learn spatio-temporal
features of varying granularities. However, aggregating events into event
frames inevitably leads to the loss of fine-grained temporal information within
frames. To remedy this drawback, we propose a novel framework termed Multi-view
Temporal Granularity aligned Aggregation (MTGA). Specifically, we first present
a novel event representation method, namely time-segmented voxel graph list,
where the most significant local voxels are temporally connected into a graph
list. Then we design a spatio-temporal fusion module based on temporal
granularity alignment, where the global spatial features extracted from event
frames, together with the local relative spatial and temporal features
contained in voxel graph list are effectively aligned and integrated. Finally,
we design a temporal aggregation module that incorporates positional encoding,
which enables the capture of local absolute spatial and global temporal
information. Experiments demonstrate that our method outperforms both the
event-based and video-based lip-reading counterparts. Our code will be publicly
available.