Structure-Aware Path Inference for Neural Finite State Transducers
Weiting Tan Chu-cheng Lin Jason Eisner
Neural finite-state transducers (NFSTs) form an expressive family of
neurosymbolic sequence transduction models. An NFST models each string pair as
having been generated by a latent path in a finite-state transducer. As they
are deep generative models, both training and inference of NFSTs require
inference networks that approximate posterior distributions over such latent
variables. In this paper, we focus on the resulting challenge of imputing the
latent alignment path that explains a given pair of input and output strings
(e.g., during training). We train three autoregressive approximate models for
amortized inference of the path, which can then be used as proposal
distributions for importance sampling. All three models perform lookahead. Our
most sophisticated (and novel) model leverages the FST structure to consider
the graph of future paths; unfortunately, we find that it loses out to the
simpler approaches -- except on an artificial task that we concocted to confuse
the simpler approaches.