Continuous Object State Recognition for Cooking Robots Using Pre-Trained
  Vision-Language Models and Black-box Optimization
Kento Kawaharazuka Naoaki Kanazawa Yoshiki Obinata Kei Okada Masayuki Inaba
The state recognition of the environment and objects by robots is generally
based on the judgement of the current state as a classification problem. On the
other hand, state changes of food in cooking happen continuously and need to be
captured not only at a certain time point but also continuously over time. In
addition, the state changes of food are complex and cannot be easily described
by manual programming. Therefore, we propose a method to recognize the
continuous state changes of food for cooking robots through the spoken language
using pre-trained large-scale vision-language models. By using models that can
compute the similarity between images and texts continuously over time, we can
capture the state changes of food while cooking. We also show that by adjusting
the weighting of each text prompt based on fitting the similarity changes to a
sigmoid function and then performing black-box optimization, more accurate and
robust continuous state recognition can be achieved. We demonstrate the
effectiveness and limitations of this method by performing the recognition of
water boiling, butter melting, egg cooking, and onion stir-frying.