Weighted Automata Extraction and Explanation of Recurrent Neural
  Networks for Natural Language Tasks
Zeming Wei Xiyue Zhang Yihao Zhang Meng Sun
Recurrent Neural Networks (RNNs) have achieved tremendous success in
processing sequential data, yet understanding and analyzing their behaviours
remains a significant challenge. To this end, many efforts have been made to
extract finite automata from RNNs, which are more amenable for analysis and
explanation. However, existing approaches like exact learning and compositional
approaches for model extraction have limitations in either scalability or
precision. In this paper, we propose a novel framework of Weighted Finite
Automata (WFA) extraction and explanation to tackle the limitations for natural
language tasks. First, to address the transition sparsity and context loss
problems we identified in WFA extraction for natural language tasks, we propose
an empirical method to complement missing rules in the transition diagram, and
adjust transition matrices to enhance the context-awareness of the WFA. We also
propose two data augmentation tactics to track more dynamic behaviours of RNN,
which further allows us to improve the extraction precision. Based on the
extracted model, we propose an explanation method for RNNs including a word
embedding method -- Transition Matrix Embeddings (TME) and TME-based task
oriented explanation for the target RNN. Our evaluation demonstrates the
advantage of our method in extraction precision than existing approaches, and
the effectiveness of TME-based explanation method in applications to
pretraining and adversarial example generation.