ACT-Net: Anchor-context Action Detection in Surgery Videos
Luoying Hao Yan Hu Wenjun Lin Qun Wang Heng Li Huazhu Fu Jinming Duan Jiang Liu
Recognition and localization of surgical detailed actions is an essential
component of developing a context-aware decision support system. However, most
existing detection algorithms fail to provide high-accuracy action classes even
having their locations, as they do not consider the surgery procedure's
regularity in the whole video. This limitation hinders their application.
Moreover, implementing the predictions in clinical applications seriously needs
to convey model confidence to earn entrustment, which is unexplored in surgical
action prediction. In this paper, to accurately detect fine-grained actions
that happen at every moment, we propose an anchor-context action detection
network (ACTNet), including an anchor-context detection (ACD) module and a
class conditional diffusion (CCD) module, to answer the following questions: 1)
where the actions happen; 2) what actions are; 3) how confidence predictions
are. Specifically, the proposed ACD module spatially and temporally highlights
the regions interacting with the extracted anchor in surgery video, which
outputs action location and its class distribution based on anchor-context
interactions. Considering the full distribution of action classes in videos,
the CCD module adopts a denoising diffusion-based generative model conditioned
on our ACD estimator to further reconstruct accurately the action predictions.
Moreover, we utilize the stochastic nature of the diffusion model outputs to
access model confidence for each prediction. Our method reports the
state-of-the-art performance, with improvements of 4.0% mAP against baseline on
the surgical video dataset.