Probing Structured Semantics Understanding and Generation of Language
  Models via Question Answering
Jinxin Liu Shulin Cao Jiaxin Shi Tingjian Zhang Lei Hou Juanzi Li
Recent advancement in the capabilities of large language models (LLMs) has
triggered a new surge in LLMs' evaluation. Most recent evaluation works tends
to evaluate the comprehensive ability of LLMs over series of tasks. However,
the deep structure understanding of natural language is rarely explored. In
this work, we examine the ability of LLMs to deal with structured semantics on
the tasks of question answering with the help of the human-constructed formal
language. Specifically, we implement the inter-conversion of natural and formal
language through in-context learning of LLMs to verify their ability to
understand and generate the structured logical forms. Extensive experiments
with models of different sizes and in different formal languages show that
today's state-of-the-art LLMs' understanding of the logical forms can approach
human level overall, but there still are plenty of room in generating correct
logical forms, which suggest that it is more effective to use LLMs to generate
more natural language training data to reinforce a small model than directly
answering questions with LLMs. Moreover, our results also indicate that models
exhibit considerable sensitivity to different formal languages. In general, the
formal language with the lower the formalization level, i.e. the more similar
it is to natural language, is more LLMs-friendly.