GelFlow: Self-supervised Learning of Optical Flow for Vision-Based
  Tactile Sensor Displacement Measurement
Zhiyuan Zhang Hua Yang Zhouping Yin
High-resolution multi-modality information acquired by vision-based tactile
sensors can support more dexterous manipulations for robot fingers. Optical
flow is low-level information directly obtained by vision-based tactile
sensors, which can be transformed into other modalities like force, geometry
and depth. Current vision-tactile sensors employ optical flow methods from
OpenCV to estimate the deformation of markers in gels. However, these methods
need to be more precise for accurately measuring the displacement of markers
during large elastic deformation of the gel, as this can significantly impact
the accuracy of downstream tasks. This study proposes a self-supervised optical
flow method based on deep learning to achieve high accuracy in displacement
measurement for vision-based tactile sensors. The proposed method employs a
coarse-to-fine strategy to handle large deformations by constructing a
multi-scale feature pyramid from the input image. To better deal with the
elastic deformation caused by the gel, the Helmholtz velocity decomposition
constraint combined with the elastic deformation constraint are adopted to
address the distortion rate and area change rate, respectively. A local flow
fusion module is designed to smooth the optical flow, taking into account the
prior knowledge of the blurred effect of gel deformation. We trained the
proposed self-supervised network using an open-source dataset and compared it
with traditional and deep learning-based optical flow methods. The results show
that the proposed method achieved the highest displacement measurement
accuracy, thereby demonstrating its potential for enabling more precise
measurement of downstream tasks using vision-based tactile sensors.