Symbol-LLM: Leverage Language Models for Symbolic System in Visual Human
  Activity Reasoning
Xiaoqian Wu Yong-Lu Li Jianhua Sun Cewu Lu
Human reasoning can be understood as a cooperation between the intuitive,
associative "System-1" and the deliberative, logical "System-2". For existing
System-1-like methods in visual activity understanding, it is crucial to
integrate System-2 processing to improve explainability, generalization, and
data efficiency. One possible path of activity reasoning is building a symbolic
system composed of symbols and rules, where one rule connects multiple symbols,
implying human knowledge and reasoning abilities. Previous methods have made
progress, but are defective with limited symbols from handcraft and limited
rules from visual-based annotations, failing to cover the complex patterns of
activities and lacking compositional generalization. To overcome the defects,
we propose a new symbolic system with two ideal important properties:
broad-coverage symbols and rational rules. Collecting massive human knowledge
via manual annotations is expensive to instantiate this symbolic system.
Instead, we leverage the recent advancement of LLMs (Large Language Models) as
an approximation of the two ideal properties, i.e., Symbols from Large Language
Models (Symbol-LLM). Then, given an image, visual contents from the images are
extracted and checked as symbols and activity semantics are reasoned out based
on rules via fuzzy logic calculation. Our method shows superiority in extensive
activity understanding tasks. Code and data are available at
https://mvig-rhos.com/symbol_llm.