The Role of Foundation Models in Neuro-Symbolic Learning and Reasoning
Daniel Cunnington Mark Law Jorge Lobo Alessandra Russo
Neuro-Symbolic AI (NeSy) holds promise to ensure the safe deployment of AI
systems, as interpretable symbolic techniques provide formal behaviour
guarantees. The challenge is how to effectively integrate neural and symbolic
computation, to enable learning and reasoning from raw data. Existing pipelines
that train the neural and symbolic components sequentially require extensive
labelling, whereas end-to-end approaches are limited in terms of scalability,
due to the combinatorial explosion in the symbol grounding problem. In this
paper, we leverage the implicit knowledge within foundation models to enhance
the performance in NeSy tasks, whilst reducing the amount of data labelling and
manual engineering. We introduce a new architecture, called NeSyGPT, which
fine-tunes a vision-language foundation model to extract symbolic features from
raw data, before learning a highly expressive answer set program to solve a
downstream task. Our comprehensive evaluation demonstrates that NeSyGPT has
superior accuracy over various baselines, and can scale to complex NeSy tasks.
Finally, we highlight the effective use of a large language model to generate
the programmatic interface between the neural and symbolic components,
significantly reducing the amount of manual engineering required.