Efficient Search and Detection of Relevant Plant Parts using
  Semantics-Aware Active Vision
Akshay K. Burusa Joost Scholten David Rapado Rincon Xin Wang Eldert J. van Henten Gert Kootstra
To automate harvesting and de-leafing of tomato plants using robots, it is
important to search and detect the relevant plant parts, namely tomatoes,
peduncles, and petioles. This is challenging due to high levels of occlusion in
tomato greenhouses. Active vision is a promising approach which helps robots to
deliberately plan camera viewpoints to overcome occlusion and improve
perception accuracy. However, current active-vision algorithms cannot
differentiate between relevant and irrelevant plant parts, making them
inefficient for targeted perception of specific plant parts. We propose a
semantic active-vision strategy that uses semantic information to identify the
relevant plant parts and prioritises them during view planning using an
attention mechanism. We evaluated our strategy using 3D models of tomato plants
with varying structural complexity, which closely represented occlusions in the
real world. We used a simulated environment to gain insights into our strategy,
while ensuring repeatability and statistical significance. At the end of ten
viewpoints, our strategy was able to correctly detect 85.5% of the plant parts,
about 4 parts more on average per plant compared to a volumetric active-vision
strategy. Also, it detected 5 and 9 parts more compared to two predefined
strategies and 11 parts more compared to a random strategy. It also performed
reliably with a median of 88.9% correctly-detected objects per plant in 96
experiments. Our strategy was also robust to uncertainty in plant and
plant-part position, plant complexity, and different viewpoint sampling
strategies. We believe that our work could significantly improve the speed and
robustness of automated harvesting and de-leafing in tomato crop production.