POTLoc: Pseudo-Label Oriented Transformer for Point-Supervised Temporal
  Action Localization
Elahe Vahdani Yingli Tian
This paper tackles the challenge of point-supervised temporal action
detection, wherein only a single frame is annotated for each action instance in
the training set. Most of the current methods, hindered by the sparse nature of
annotated points, struggle to effectively represent the continuous structure of
actions or the inherent temporal and semantic dependencies within action
instances. Consequently, these methods frequently learn merely the most
distinctive segments of actions, leading to the creation of incomplete action
proposals. This paper proposes POTLoc, a Pseudo-label Oriented Transformer for
weakly-supervised Action Localization utilizing only point-level annotation.
POTLoc is designed to identify and track continuous action structures via a
self-training strategy. The base model begins by generating action proposals
solely with point-level supervision. These proposals undergo refinement and
regression to enhance the precision of the estimated action boundaries, which
subsequently results in the production of `pseudo-labels' to serve as
supplementary supervisory signals. The architecture of the model integrates a
transformer with a temporal feature pyramid to capture video snippet
dependencies and model actions of varying duration. The pseudo-labels,
providing information about the coarse locations and boundaries of actions,
assist in guiding the transformer for enhanced learning of action dynamics.
POTLoc outperforms the state-of-the-art point-supervised methods on THUMOS'14
and ActivityNet-v1.2 datasets, showing a significant improvement of 5% average
mAP on the former.