Unlocking the Transferability of Tokens in Deep Models for Tabular Data
Qi-Le Zhou Han-Jia Ye Le-Ye Wang De-Chuan Zhan
Fine-tuning a pre-trained deep neural network has become a successful
paradigm in various machine learning tasks. However, such a paradigm becomes
particularly challenging with tabular data when there are discrepancies between
the feature sets of pre-trained models and the target tasks. In this paper, we
propose TabToken, a method aims at enhancing the quality of feature tokens
(i.e., embeddings of tabular features). TabToken allows for the utilization of
pre-trained models when the upstream and downstream tasks share overlapping
features, facilitating model fine-tuning even with limited training examples.
Specifically, we introduce a contrastive objective that regularizes the tokens,
capturing the semantics within and across features. During the pre-training
stage, the tokens are learned jointly with top-layer deep models such as
transformer. In the downstream task, tokens of the shared features are kept
fixed while TabToken efficiently fine-tunes the remaining parts of the model.
TabToken not only enables knowledge transfer from a pre-trained model to tasks
with heterogeneous features, but also enhances the discriminative ability of
deep tabular models in standard classification and regression tasks.