Deformable Convolutions and LSTM-based Flexible Event Frame Fusion
  Network for Motion Deblurring
Dan Yang Mehmet Yamac
Event cameras differ from conventional RGB cameras in that they produce
asynchronous data sequences. While RGB cameras capture every frame at a fixed
rate, event cameras only capture changes in the scene, resulting in sparse and
asynchronous data output. Despite the fact that event data carries useful
information that can be utilized in motion deblurring of RGB cameras,
integrating event and image information remains a challenge. Recent
state-of-the-art CNN-based deblurring solutions produce multiple 2-D event
frames based on the accumulation of event data over a time period. In most of
these techniques, however, the number of event frames is fixed and predefined,
which reduces temporal resolution drastically, particularly for scenarios when
fast-moving objects are present or when longer exposure times are required. It
is also important to note that recent modern cameras (e.g., cameras in mobile
phones) dynamically set the exposure time of the image, which presents an
additional problem for networks developed for a fixed number of event frames. A
Long Short-Term Memory (LSTM)-based event feature extraction module has been
developed for addressing these challenges, which enables us to use a
dynamically varying number of event frames. Using these modules, we constructed
a state-of-the-art deblurring network, Deformable Convolutions and LSTM-based
Flexible Event Frame Fusion Network (DLEFNet). It is particularly useful for
scenarios in which exposure times vary depending on factors such as lighting
conditions or the presence of fast-moving objects in the scene. It has been
demonstrated through evaluation results that the proposed method can outperform
the existing state-of-the-art networks for deblurring task in synthetic and
real-world data sets.