Adaptive User-centered Neuro-symbolic Learning for Multimodal
  Interaction with Autonomous Systems
Amr Gomaa Michael Feld
Recent advances in machine learning, particularly deep learning, have enabled
autonomous systems to perceive and comprehend objects and their environments in
a perceptual subsymbolic manner. These systems can now perform object
detection, sensor data fusion, and language understanding tasks. However, there
is a growing need to enhance these systems to understand objects and their
environments more conceptually and symbolically. It is essential to consider
both the explicit teaching provided by humans (e.g., describing a situation or
explaining how to act) and the implicit teaching obtained by observing human
behavior (e.g., through the system's sensors) to achieve this level of powerful
artificial intelligence. Thus, the system must be designed with multimodal
input and output capabilities to support implicit and explicit interaction
models. In this position paper, we argue for considering both types of inputs,
as well as human-in-the-loop and incremental learning techniques, for advancing
the field of artificial intelligence and enabling autonomous systems to learn
like humans. We propose several hypotheses and design guidelines and highlight
a use case from related work to achieve this goal.