A Video-Based Activity Classification of Human Pickers in Agriculture
Abhishesh Pal Antonio C. Leite Jon G. O. Gjevestad Pål J. From
In farming systems, harvesting operations are tedious, time- and
resource-consuming tasks. Based on this, deploying a fleet of autonomous robots
to work alongside farmworkers may provide vast productivity and logistics
benefits. Then, an intelligent robotic system should monitor human behavior,
identify the ongoing activities and anticipate the worker's needs. In this
work, the main contribution consists of creating a benchmark model for
video-based human pickers detection, classifying their activities to serve in
harvesting operations for different agricultural scenarios. Our solution uses
the combination of a Mask Region-based Convolutional Neural Network (Mask
R-CNN) for object detection and optical flow for motion estimation with newly
added statistical attributes of flow motion descriptors, named as Correlation
Sensitivity (CS). A classification criterion is defined based on the Kernel
Density Estimation (KDE) analysis and K-means clustering algorithm, which are
implemented upon in-house collected dataset from different crop fields like
strawberry polytunnels and apple tree orchards. The proposed framework is
quantitatively analyzed using sensitivity, specificity, and accuracy measures
and shows satisfactory results amidst various dataset challenges such as
lighting variation, blur, and occlusions.