The KANDY Benchmark: Incremental Neuro-Symbolic Learning and Reasoning
  with Kandinsky Patterns
Luca Salvatore Lorello Marco Lippi Stefano Melacci
Artificial intelligence is continuously seeking novel challenges and
benchmarks to effectively measure performance and to advance the
state-of-the-art. In this paper we introduce KANDY, a benchmarking framework
that can be used to generate a variety of learning and reasoning tasks inspired
by Kandinsky patterns. By creating curricula of binary classification tasks
with increasing complexity and with sparse supervisions, KANDY can be used to
implement benchmarks for continual and semi-supervised learning, with a
specific focus on symbol compositionality. Classification rules are also
provided in the ground truth to enable analysis of interpretable solutions.
Together with the benchmark generation pipeline, we release two curricula, an
easier and a harder one, that we propose as new challenges for the research
community. With a thorough experimental evaluation, we show how both
state-of-the-art neural models and purely symbolic approaches struggle with
solving most of the tasks, thus calling for the application of advanced
neuro-symbolic methods trained over time.