Three Pathways to Neurosymbolic Reinforcement Learning with
  Interpretable Model and Policy Networks
Peter Graf Patrick Emami
Neurosymbolic AI combines the interpretability, parsimony, and explicit
reasoning of classical symbolic approaches with the statistical learning of
data-driven neural approaches. Models and policies that are simultaneously
differentiable and interpretable may be key enablers of this marriage. This
paper demonstrates three pathways to implementing such models and policies in a
real-world reinforcement learning setting. Specifically, we study a broad class
of neural networks that build interpretable semantics directly into their
architecture. We reveal and highlight both the potential and the essential
difficulties of combining logic, simulation, and learning. One lesson is that
learning benefits from continuity and differentiability, but classical logic is
discrete and non-differentiable. The relaxation to real-valued, differentiable
representations presents a trade-off; the more learnable, the less
interpretable. Another lesson is that using logic in the context of a numerical
simulation involves a non-trivial mapping from raw (e.g., real-valued time
series) simulation data to logical predicates. Some open questions this note
exposes include: What are the limits of rule-based controllers, and how
learnable are they? Do the differentiable interpretable approaches discussed
here scale to large, complex, uncertain systems? Can we truly achieve
interpretability? We highlight these and other themes across the three
approaches.