Introducing Hybrid Modeling with Time-series-Transformers: A Comparative
  Study of Series and Parallel Approach in Batch Crystallization
Niranjan Sitapure Joseph S Kwon
Most existing digital twins rely on data-driven black-box models,
predominantly using deep neural recurrent, and convolutional neural networks
(DNNs, RNNs, and CNNs) to capture the dynamics of chemical systems. However,
these models have not seen the light of day, given the hesitance of directly
deploying a black-box tool in practice due to safety and operational issues. To
tackle this conundrum, hybrid models combining first-principles physics-based
dynamics with machine learning (ML) models have increased in popularity as they
are considered a 'best of both worlds' approach. That said, existing simple DNN
models are not adept at long-term time-series predictions and utilizing
contextual information on the trajectory of the process dynamics. Recently,
attention-based time-series transformers (TSTs) that leverage multi-headed
attention mechanism and positional encoding to capture long-term and short-term
changes in process states have shown high predictive performance. Thus, a
first-of-a-kind, TST-based hybrid framework has been developed for batch
crystallization, demonstrating improved accuracy and interpretability compared
to traditional black-box models. Specifically, two different configurations
(i.e., series and parallel) of TST-based hybrid models are constructed and
compared, which show a normalized-mean-square-error (NMSE) in the range of
$[10, 50]\times10^{-4}$ and an $R^2$ value over 0.99. Given the growing
adoption of digital twins, next-generation attention-based hybrid models are
expected to play a crucial role in shaping the future of chemical
manufacturing.