DAC-MR: Data Augmentation Consistency Based Meta-Regularization for
  Meta-Learning
Jun Shu Xiang Yuan Deyu Meng Zongben Xu
Meta learning recently has been heavily researched and helped advance the
contemporary machine learning. However, achieving well-performing meta-learning
model requires a large amount of training tasks with high-quality meta-data
representing the underlying task generalization goal, which is sometimes
difficult and expensive to obtain for real applications. Current
meta-data-driven meta-learning approaches, however, are fairly hard to train
satisfactory meta-models with imperfect training tasks. To address this issue,
we suggest a meta-knowledge informed meta-learning (MKIML) framework to improve
meta-learning by additionally integrating compensated meta-knowledge into
meta-learning process. We preliminarily integrate meta-knowledge into
meta-objective via using an appropriate meta-regularization (MR) objective to
regularize capacity complexity of the meta-model function class to facilitate
better generalization on unseen tasks. As a practical implementation, we
introduce data augmentation consistency to encode invariance as meta-knowledge
for instantiating MR objective, denoted by DAC-MR. The proposed DAC-MR is
hopeful to learn well-performing meta-models from training tasks with noisy,
sparse or unavailable meta-data. We theoretically demonstrate that DAC-MR can
be treated as a proxy meta-objective used to evaluate meta-model without
high-quality meta-data. Besides, meta-data-driven meta-loss objective combined
with DAC-MR is capable of achieving better meta-level generalization. 10
meta-learning tasks with different network architectures and benchmarks
substantiate the capability of our DAC-MR on aiding meta-model learning. Fine
performance of DAC-MR are obtained across all settings, and are well-aligned
with our theoretical insights. This implies that our DAC-MR is
problem-agnostic, and hopeful to be readily applied to extensive meta-learning
problems and tasks.