Dietary Assessment with Multimodal ChatGPT: A Systematic Analysis
Frank P. -W. Lo Jianing Qiu Zeyu Wang Junhong Chen Bo Xiao Wu Yuan Stamatia Giannarou Gary Frost Benny Lo
Conventional approaches to dietary assessment are primarily grounded in
self-reporting methods or structured interviews conducted under the supervision
of dietitians. These methods, however, are often subjective, potentially
inaccurate, and time-intensive. Although artificial intelligence (AI)-based
solutions have been devised to automate the dietary assessment process, these
prior AI methodologies encounter challenges in their ability to generalize
across a diverse range of food types, dietary behaviors, and cultural contexts.
This results in AI applications in the dietary field that possess a narrow
specialization and limited accuracy. Recently, the emergence of multimodal
foundation models such as GPT-4V powering the latest ChatGPT has exhibited
transformative potential across a wide range of tasks (e.g., Scene
understanding and image captioning) in numerous research domains. These models
have demonstrated remarkable generalist intelligence and accuracy, capable of
processing various data modalities. In this study, we explore the application
of multimodal ChatGPT within the realm of dietary assessment. Our findings
reveal that GPT-4V excels in food detection under challenging conditions with
accuracy up to 87.5% without any fine-tuning or adaptation using food-specific
datasets. By guiding the model with specific language prompts (e.g., African
cuisine), it shifts from recognizing common staples like rice and bread to
accurately identifying regional dishes like banku and ugali. Another GPT-4V's
standout feature is its contextual awareness. GPT-4V can leverage surrounding
objects as scale references to deduce the portion sizes of food items, further
enhancing its accuracy in translating food weight into nutritional content.
This alignment with the USDA National Nutrient Database underscores GPT-4V's
potential to advance nutritional science and dietary assessment techniques.