Learning a Decision Tree Algorithm with Transformers
Yufan Zhuang Liyuan Liu Chandan Singh Jingbo Shang Jianfeng Gao
Decision trees are renowned for their interpretability capability to achieve
high predictive performance, especially on tabular data. Traditionally, they
are constructed through recursive algorithms, where they partition the data at
every node in a tree. However, identifying the best partition is challenging,
as decision trees optimized for local segments may not bring global
generalization. To address this, we introduce MetaTree, which trains a
transformer-based model on filtered outputs from classical algorithms to
produce strong decision trees for classification. Specifically, we fit both
greedy decision trees and optimized decision trees on a large number of
datasets. We then train MetaTree to produce the trees that achieve strong
generalization performance. This training enables MetaTree to not only emulate
these algorithms, but also to intelligently adapt its strategy according to the
context, thereby achieving superior generalization performance.