FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data
Dongyu Zhang Liang Wang Xin Dai Shubham Jain Junpeng Wang Yujie Fan Chin-Chia Michael Yeh Yan Zheng Zhongfang Zhuang Wei Zhang
Sequential tabular data is one of the most commonly used data types in
real-world applications. Different from conventional tabular data, where rows
in a table are independent, sequential tabular data contains rich contextual
and sequential information, where some fields are dynamically changing over
time and others are static. Existing transformer-based approaches analyzing
sequential tabular data overlook the differences between dynamic and static
fields by replicating and filling static fields into each transformer, and
ignore temporal information between rows, which leads to three major
disadvantages: (1) computational overhead, (2) artificially simplified data for
masked language modeling pre-training task that may yield less meaningful
representations, and (3) disregarding the temporal behavioral patterns implied
by time intervals. In this work, we propose FATA-Trans, a model with two field
transformers for modeling sequential tabular data, where each processes static
and dynamic field information separately. FATA-Trans is field- and time-aware
for sequential tabular data. The field-type embedding in the method enables
FATA-Trans to capture differences between static and dynamic fields. The
time-aware position embedding exploits both order and time interval information
between rows, which helps the model detect underlying temporal behavior in a
sequence. Our experiments on three benchmark datasets demonstrate that the
learned representations from FATA-Trans consistently outperform
state-of-the-art solutions in the downstream tasks. We also present
visualization studies to highlight the insights captured by the learned
representations, enhancing our understanding of the underlying data. Our codes
are available at https://github.com/zdy93/FATA-Trans.