MotherNet: A Foundational Hypernetwork for Tabular Classification
Andreas Müller Carlo Curino Raghu Ramakrishnan
The advent of Foundation Models is transforming machine learning across many
modalities (e.g., language, images, videos) with prompt engineering replacing
training in many settings. Recent work on tabular data (e.g., TabPFN) hints at
a similar opportunity to build Foundation Models for classification for
numerical data. In this paper, we go one step further and propose a
hypernetwork architecture that we call MotherNet, trained on millions of
classification tasks, that, once prompted with a never-seen-before training set
generates the weights of a trained ``child'' neural-network. Like other
Foundation Models, MotherNet replaces training on specific datasets with
in-context learning through a single forward pass. In contrast to existing
hypernetworks that were either task-specific or trained for relatively
constraint multi-task settings, MotherNet is trained to generate networks to
perform multiclass classification on arbitrary tabular datasets without any
dataset specific gradient descent.
  The child network generated by MotherNet using in-context learning
outperforms neural networks trained using gradient descent on small datasets,
and is competitive with predictions by TabPFN and standard ML methods like
Gradient Boosting. Unlike a direct application of transformer models like
TabPFN, MotherNet generated networks are highly efficient at inference time.
This methodology opens up a new approach to building predictive models on
tabular data that is both efficient and robust, without any dataset-specific
training.