Boosting-Based Sequential Meta-Tree Ensemble Construction for Improved
  Decision Trees
Ryota Maniwa Naoki Ichijo Yuta Nakahara Toshiyasu Matsushima
A decision tree is one of the most popular approaches in machine learning
fields. However, it suffers from the problem of overfitting caused by overly
deepened trees. Then, a meta-tree is recently proposed. It solves the problem
of overfitting caused by overly deepened trees. Moreover, the meta-tree
guarantees statistical optimality based on Bayes decision theory. Therefore,
the meta-tree is expected to perform better than the decision tree. In contrast
to a single decision tree, it is known that ensembles of decision trees, which
are typically constructed boosting algorithms, are more effective in improving
predictive performance. Thus, it is expected that ensembles of meta-trees are
more effective in improving predictive performance than a single meta-tree, and
there are no previous studies that construct multiple meta-trees in boosting.
Therefore, in this study, we propose a method to construct multiple meta-trees
using a boosting approach. Through experiments with synthetic and benchmark
datasets, we conduct a performance comparison between the proposed methods and
the conventional methods using ensembles of decision trees. Furthermore, while
ensembles of decision trees can cause overfitting as well as a single decision
tree, experiments confirmed that ensembles of meta-trees can prevent
overfitting due to the tree depth.