Adapting deep networks to new concepts from a few examples is challenging, due to the high computational requirements of standard fine-tuning procedures.
 Most work on few-shot learning has thus focused on simple learning techniques for adaptation, such as nearest neighbours or gradient descent.
 Nonetheless, the machine learning literature contains a wealth of methods that learn non-deep models very efficiently.
 In this paper, we propose to use these fast convergent methods as the main adaptation mechanism for few-shot learning.
 The main idea is to teach a deep network to use standard machine learning tools, such as ridge regression, as part of its own internal model, enabling it to quickly adapt to novel data.
 This requires back-propagating errors through the solver steps.
 While normally the cost of the matrix operations involved in such a process would be significant, by using the Woodbury identity we can make the small number of examples work to our advantage.
 We propose both closed-form and iterative solvers, based on ridge regression and logistic regression components.
 Our methods constitute a simple and novel approach to the problem of few-shot learning and achieve performance competitive with or superior to the state of the art on three benchmarks. Humans can efficiently perform fast mapping BID7 BID8 , i.e. learning a new concept after a single exposure. By contrast, supervised learning algorithms -and neural networks in particular -typically need to be trained using a vast amount of data in order to generalize well. This requirement is problematic, as the availability of large labelled datasets cannot always be taken for granted. Labels can be costly to acquire: in drug discovery, for instance, campaign budgets often limits researchers to only operate with a small amount of biological data that can be used to form predictions about properties and activities of compounds BID0 . In other circumstances, data itself can be scarce, as it can happen for example with the problem of classifying rare animal species, whose exemplars are not easy to observe. Such a scenario, in which just one or a handful of training examples is provided, is referred to as one-shot or few-shot learning BID28 BID12 BID25 BID18 and has recently seen a tremendous surge in interest within the machine learning community (e.g. ; BID4 ; BID39 ; BID14 ).Currently , most methods tackling few-shot learning operate within the general paradigm of metalearning, which allows one to develop algorithms in which the process of learning can improve with the number of training episodes BID53 BID57 . This can be achieved by distilling and transferring knowledge across episodes. In practice , for the problem of few-shot classification, meta-learning is often implemented using two "nested training loops". The base learner works at the level of individual episodes, which correspond to learning problems characterised by having only a small set of labelled training images available. The meta learner , by contrast, learns from a collection of such episodes, with the goal of improving the performance of the base learner across episodes. Episode N Figure 1 : Diagram of the proposed method for one episode, of which several are seen during meta-training. The task is to learn new classes given just a few sample images per class. In this illustrative example, there are 3 classes and 2 samples per class, making each episode a 3-way, 2-shot classification problem. At the base learning level, learning is accomplished by a differentiable ridge regression layer (R.R.), which computes episode-specific weights (referred to as w E in Section 3.1 and as W in Section 3.2). At the meta-training level, by back-propagating errors through many of these small learning problems, we train a network whose weights are shared across episodes, together with the hyper-parameters of the R.R. layer. In this way, the R.R. base learner can improve its learning capabilities as the number of experienced episodes increases.Clearly, in any meta-learning algorithm, it is of paramount importance to choose the base learner carefully. On one side of the spectrum , methods related to nearest-neighbours, such as learning similarity functions BID23 BID48 , are fast but rely solely on the quality of the similarity metric, with no additional data-dependent adaptation at test-time. On the other side of the spectrum , methods that optimize standard iterative learning algorithms, such as backpropagating through gradient descent BID14 BID35 or explicitly learning the learner's update rule BID1 BID39 , are slower but allow more adaptability to different problems/datasets.In this paper, we take a different perspective. As base learners, we propose to adopt simple learning algorithms that admit a closed-form solution such as ridge regression. Crucially, the simplicity and differentiability of these solutions allow us to backpropagate through learning problems. Moreover, these algorithms are particularly suitable for use within a meta-learning framework for few-shot classification for two main reasons. First, their closed-form solution allows learning problems to be solved efficiently. Second, in a data regime characterized by few examples of high dimensionality, the Woodbury's identity (Petersen et al., 2008, Chapter 3.2) can be used to obtain a very significant gain in terms of computational speed.We demonstrate the strength of our approach by performing extensive experiments on Omniglot (Lake et al., 2015), CIFAR-100 BID24 ) (adapted to the few-shot problem) and miniImageNet . Our base learners are fast, simple to implement, and can achieve performance that is competitive with or superior to the state of the art in terms of accuracy. With the aim of allowing efficient adaptation to unseen learning problems, in this paper we explored the feasibility of incorporating fast solvers with closed-form solutions as the base learning component of a meta-learning system. Importantly, the use of the Woodbury identity allows significant computational gains in a scenario presenting only a few samples with high dimensionality, like one-shot of few-shot learning. R2-D2, the differentiable ridge regression base learner we introduce, is almost as fast as prototypical networks and strikes a useful compromise between not performing adaptation for new episodes (like metric-learning-based approaches) and conducting a costly iterative approach (like MAML or LSTM-based meta-learners). In general, we showed that our base learners work remarkably well, with excellent results on few-shot learning benchmarks, generalizing to episodes with new classes that were not seen during training. We believe that our findings point in an exciting direction of more sophisticated yet efficient online adaptation methods, able to leverage the potential of prior knowledge distilled in an offline training phase. In future work, we would like to explore Newton's methods with more complicated second-order structure than ridge regression. Contributions within the few-shot learning paradigm. In this work, we evaluated our proposed methods R2-D2 and LR-D2 in the few-shot learning scenario BID12 BID25 BID39 BID18 , which consists in learning how to discriminate between images given one or very few examples. For methods tackling this problem, it is common practice to organise the training procedure in two nested loops. The inner loop is used to solve the actual few-shot classification problem, while the outer loop serves as a guidance for the former by gradually modifying the inductive bias of the base learner BID57 . Differently from standard classification benchmarks, the few-shot ones enforce that classes are disjoint between dataset splits.In the literature (e.g. ), the very small classification problems with unseen classes solved within the inner loop have often been referred to as episodes or tasks. Considering the general few-shot learning paradigm just described, methods in the recent literature mostly differ for the type of learner they use in the inner loop and the amount of per-episode adaptability they allow. For example, at the one end of the spectrum in terms of "amount of adaptability", we can find methods such as MAML Finn et al. (2017) , which learns how to efficiently fine-tune the parameters of a neural-network with few iterations of SGD. On the other end, we have methods based on metric learning such as prototypical networks BID48 and relation network BID50 , which are fast but do not perform adaptation. Note that the amount of adaptation to a new episode (i.e.a new classification problem with unseen classes) is not at all indicative of the performance in few-shot learning benchmarks. As a matter of fact, both BID48 and BID50 achieve higher accuracy than MAML. Nonetheless, adaptability is a desirable property, as it allows more design flexibility.Within this landscape, our work proposes a novel technique (R2-D2) that does allow per-episode adaptation while at the same time being fast TAB6 ) and achieving strong performance TAB0 . The key innovation is to use a simple (and differentiable) solver such as ridge regression within the inner loop, which requires back-propagating through the solution of a learning problem. Crucially, its closed-form solution and the use of the Woodbury identity (particularly advantageous in the low data regime) allow this non-trivial endeavour to be efficient. We further demonstrate that this strategy is not limited to the ridge regression case, but it can also be extended to other solvers (LR-D2) by dividing the problem into a short series of weighted least squares problems ( (Murphy, 2012, Chapter 8.3.4
We propose a meta-learning approach for few-shot classification that achieves strong performance at high-speed by back-propagating through the solution of fast solvers, such as ridge regression or logistic regression. The paper proposes an algorithm for meta-learning which amounts to fixing the features (ie all hidden layers of a deep NN), and treating each task as having its own final layer which could be a ridge regression or a logistic regression. This paper proposes a meta-learning approach for the problem of few-shot classification, they use a  method based on parametrizing the learner for each task by a closed-form solver.