Foundations for Transfer in Reinforcement Learning: A Taxonomy of
  Knowledge Modalities
Markus Wulfmeier Arunkumar Byravan Sarah Bechtle Karol Hausman Nicolas Heess
Contemporary artificial intelligence systems exhibit rapidly growing
abilities accompanied by the growth of required resources, expansive datasets
and corresponding investments into computing infrastructure. Although earlier
successes predominantly focus on constrained settings, recent strides in
fundamental research and applications aspire to create increasingly general
systems. This evolving landscape presents a dual panorama of opportunities and
challenges in refining the generalisation and transfer of knowledge - the
extraction from existing sources and adaptation as a comprehensive foundation
for tackling new problems. Within the domain of reinforcement learning (RL),
the representation of knowledge manifests through various modalities, including
dynamics and reward models, value functions, policies, and the original data.
This taxonomy systematically targets these modalities and frames its discussion
based on their inherent properties and alignment with different objectives and
mechanisms for transfer. Where possible, we aim to provide coarse guidance
delineating approaches which address requirements such as limiting environment
interactions, maximising computational efficiency, and enhancing generalisation
across varying axes of change. Finally, we analyse reasons contributing to the
prevalence or scarcity of specific forms of transfer, the inherent potential
behind pushing these frontiers, and underscore the significance of
transitioning from designed to learned transfer.