FIRE: Food Image to REcipe generation
Prateek Chhikara Dhiraj Chaurasia Yifan Jiang Omkar Masur Filip Ilievski
Food computing has emerged as a prominent multidisciplinary field of research
in recent years. An ambitious goal of food computing is to develop end-to-end
intelligent systems capable of autonomously producing recipe information for a
food image. Current image-to-recipe methods are retrieval-based and their
success depends heavily on the dataset size and diversity, as well as the
quality of learned embeddings. Meanwhile, the emergence of powerful
attention-based vision and language models presents a promising avenue for
accurate and generalizable recipe generation, which has yet to be extensively
explored. This paper proposes FIRE, a novel multimodal methodology tailored to
recipe generation in the food computing domain, which generates the food title,
ingredients, and cooking instructions based on input food images. FIRE
leverages the BLIP model to generate titles, utilizes a Vision Transformer with
a decoder for ingredient extraction, and employs the T5 model to generate
recipes incorporating titles and ingredients as inputs. We showcase two
practical applications that can benefit from integrating FIRE with large
language model prompting: recipe customization to fit recipes to user
preferences and recipe-to-code transformation to enable automated cooking
processes. Our experimental findings validate the efficacy of our proposed
approach, underscoring its potential for future advancements and widespread
adoption in food computing.