Visual AI and Linguistic Intelligence Through Steerability and
  Composability
David Noever Samantha Elizabeth Miller Noever
This study explores the capabilities of multimodal large language models
(LLMs) in handling challenging multistep tasks that integrate language and
vision, focusing on model steerability, composability, and the application of
long-term memory and context understanding. The problem addressed is the LLM's
ability (Nov 2023 GPT-4 Vision Preview) to manage tasks that require
synthesizing visual and textual information, especially where stepwise
instructions and sequential logic are paramount. The research presents a series
of 14 creatively and constructively diverse tasks, ranging from AI Lego
Designing to AI Satellite Image Analysis, designed to test the limits of
current LLMs in contexts that previously proved difficult without extensive
memory and contextual understanding. Key findings from evaluating 800 guided
dialogs include notable disparities in task completion difficulty. For
instance, 'Image to Ingredient AI Bartender' (Low difficulty) contrasted
sharply with 'AI Game Self-Player' (High difficulty), highlighting the LLM's
varying proficiency in processing complex visual data and generating coherent
instructions. Tasks such as 'AI Genetic Programmer' and 'AI Negotiator' showed
high completion difficulty, emphasizing challenges in maintaining context over
multiple steps. The results underscore the importance of developing LLMs that
combine long-term memory and contextual awareness to mimic human-like thought
processes in complex problem-solving scenarios.