Spatiotemporal Event Graphs for Dynamic Scene Understanding
Salman Khan
Dynamic scene understanding is the ability of a computer system to interpret
and make sense of the visual information present in a video of a real-world
scene. In this thesis, we present a series of frameworks for dynamic scene
understanding starting from road event detection from an autonomous driving
perspective to complex video activity detection, followed by continual learning
approaches for the life-long learning of the models. Firstly, we introduce the
ROad event Awareness Dataset (ROAD) for Autonomous Driving, to our knowledge
the first of its kind. Due to the lack of datasets equipped with formally
specified logical requirements, we also introduce the ROad event Awareness
Dataset with logical Requirements (ROAD-R), the first publicly available
dataset for autonomous driving with requirements expressed as logical
constraints, as a tool for driving neurosymbolic research in the area. Next, we
extend event detection to holistic scene understanding by proposing two complex
activity detection methods. In the first method, we present a deformable,
spatiotemporal scene graph approach, consisting of three main building blocks:
action tube detection, a 3D deformable RoI pooling layer designed for learning
the flexible, deformable geometry of the constituent action tubes, and a scene
graph constructed by considering all parts as nodes and connecting them based
on different semantics. In a second approach evolving from the first, we
propose a hybrid graph neural network that combines attention applied to a
graph encoding of the local (short-term) dynamic scene with a temporal graph
modelling the overall long-duration activity. Finally, the last part of the
thesis is about presenting a new continual semi-supervised learning (CSSL)
paradigm.