Statistical Foundations of Prior-Data Fitted Networks
Thomas Nagler
Prior-data fitted networks (PFNs) were recently proposed as a new paradigm
for machine learning. Instead of training the network to an observed training
set, a fixed model is pre-trained offline on small, simulated training sets
from a variety of tasks. The pre-trained model is then used to infer class
probabilities in-context on fresh training sets with arbitrary size and
distribution. Empirically, PFNs achieve state-of-the-art performance on tasks
with similar size to the ones used in pre-training. Surprisingly, their
accuracy further improves when passed larger data sets during inference. This
article establishes a theoretical foundation for PFNs and illuminates the
statistical mechanisms governing their behavior. While PFNs are motivated by
Bayesian ideas, a purely frequentistic interpretation of PFNs as pre-tuned, but
untrained predictors explains their behavior. A predictor's variance vanishes
if its sensitivity to individual training samples does and the bias vanishes
only if it is appropriately localized around the test feature. The transformer
architecture used in current PFN implementations ensures only the former. These
findings shall prove useful for designing architectures with favorable
empirical behavior.