Team I2R-VI-FF Technical Report on EPIC-KITCHENS VISOR Hand Object
  Segmentation Challenge 2023
Fen Fang Yi Cheng Ying Sun Qianli Xu
In this report, we present our approach to the EPIC-KITCHENS VISOR Hand
Object Segmentation Challenge, which focuses on the estimation of the relation
between the hands and the objects given a single frame as input. The
EPIC-KITCHENS VISOR dataset provides pixel-wise annotations and serves as a
benchmark for hand and active object segmentation in egocentric video. Our
approach combines the baseline method, i.e., Point-based Rendering (PointRend)
and the Segment Anything Model (SAM), aiming to enhance the accuracy of hand
and object segmentation outcomes, while also minimizing instances of missed
detection. We leverage accurate hand segmentation maps obtained from the
baseline method to extract more precise hand and in-contact object segments. We
utilize the class-agnostic segmentation provided by SAM and apply specific
hand-crafted constraints to enhance the results. In cases where the baseline
model misses the detection of hands or objects, we re-train an object detector
on the training set to enhance the detection accuracy. The detected hand and
in-contact object bounding boxes are then used as prompts to extract their
respective segments from the output of SAM. By effectively combining the
strengths of existing methods and applying our refinements, our submission
achieved the 1st place in terms of evaluation criteria in the VISOR HOS
Challenge.