Recent deep multi-task learning (MTL) has been witnessed its success in alleviating data scarcity of some task by utilizing domain-specific knowledge from related tasks. Nonetheless, several major issues of deep MTL, including the effectiveness of sharing mechanisms, the efficiency of model complexity and the flexibility of network architectures, still remain largely unaddressed. To this end, we propose a novel generalized latent-subspace based knowledge sharing mechanism for linking task-specific models, namely tensor ring multi-task learning (TRMTL). TRMTL has a highly compact representation, and it is very effective in transferring task-invariant knowledge while being super flexible in learning task-specific features, successfully mitigating the dilemma of both negative-transfer in lower layers and under-transfer in higher layers. Under our TRMTL, it is feasible for each task to have heterogenous input data dimensionality or distinct feature sizes at different hidden layers. Experiments on a variety of datasets demonstrate our model is capable of significantly improving each single task’s performance, particularly favourable in scenarios where some of the tasks have insufficient data. Multi-task learning (MTL) (Caruana, 1997; BID15 is an approach for boosting the overall performance in each individual task by learning multiple related tasks simultaneously. In the deep learning context, jointly fitting sufficiently flexible deep neural networks (DNNs) to data of multiple tasks can be seen as adding an inductive bias to the deep models, which could be beneficial to learn feature representations preferable by all tasks. Recently, the deep MTL has gained much popularity and been successfully explored in an abroad range of applications, such as computer vision BID32 BID16 , natural language processing BID14 , speech recognition BID5 and so on.However, a number of key challenges posed by the issues of ineffectiveness, inefficiency and inflexibility in deep MTL are left largely unaddressed. One major challenge is how to seek effective information sharing mechanisms across related tasks, which is equivalent to designing better parameter sharing patterns in the deep networks. Some previous work BID32 BID30 tried to solve this problem by means of hard parameter sharing BID21 , where the bottom layers are all shared except with one branch per task at the top layers. Although being simple and robust to over-fitting BID0 , this kind of architecture can be harmful when learning high-level task-specific features, since it focuses only on common low-level features of all tasks. Moreover, these common features may be polluted by some noxious tasks, leading to the negative transfer in low-level features among tasks BID31 ). An alternative line of work mitigate this issue to some extent by following the soft parameter sharing strategy BID21 , under which one separate DNN is learned for each task with its own set of parameters, and the individual DNNs are implicitly linked by imposing constraints on the aligned weights. The deep MTL models of this type include using 2 norm regularization BID4 , trace norm regularization BID29 and tensor norm priors BID12 BID13 .The lack of efficiency in model complexity gives rise to another great challenge for current deep MTL. The above soft-sharing based deep models (one set of parameters per task) typically involve enormous number of trainable parameters and require extremely large storage and memory. It is thus usually infeasible to deploy those deep MTL models on resource-constrained devices such as mobile The overall sharing mechanisms of MRN, two variants of DMTRL (for the setting of CNN) and our TRMTL w.r.t. two tasks. The shared portion is depicted in yellow. The circles, squares and thin rectangles represent tensor cores, matrices and vectors, respectively. MRN : original weights are totally shared at the lower layers and the relatedness between tasks at the top layers is modeled by tensor normal priors. DMTRL (TT or Tucker): all layer-wise weights must be equal-sized so as to be stacked and decomposed into factors. For each task, almost all the factors are shard at each layer except the very last 1D vector. Such pattern of sharing is identical at all layers. TRMTL: layer-wise weights are separately encoded into TR-formats for different tasks, and a subset of latent cores are selected to be tied across two tasks. The portions of sharing can be different from layer to layer. phones and wearable computers . BID28 alleviated the issue by integrating tensor factorization with deep MTL and proposed deep multi-task representation learning (DMTRL). Specifically, they first stack up the layer-wise weights from all tasks and then decompose them into low-rank factors, yielding a succinct deep MTL model with fewer parameters. Despite the compactness of the model, DMTRL turns out to be rather restricted on sharing knowledge effectively. This is because, as shown in FIG0 , DMTRL (TT or Tucker) shares almost all fractions of layer-wise weights as common factors, leaving only a tiny portion of weights to encode the task-specific information. Even worse, such pattern of sharing must be identical across all hidden layers, which is vulnerable to the negative transfer of the features. As an effect, the common factors become highly dominant at each layer and greatly suppress model's capability in expressing task-specific variations.The last challenge arises from the flexibility of architecture in deep MTL. Most of deep MTL models force tasks to have the equal-sized layer-wise weights or input dimensionality. This restriction makes little sense for the case of loosely-related tasks, since individual tasks' features (input data) can be quite different and the sizes of layer-wise features (input data) may vary a lot from task to task.In this work, we provide a generalized latent-subspace based solution to addressing aforementioned difficulties of deep MTL, from all aspects of effectiveness, efficiency and flexibility. Regarding the effectiveness, we propose to share different portions of weights as common knowledge at distinct layers, so that each individual task can better convey its private knowledge. As for the efficiency, our proposal shares knowledge in the latent subspace instead of original space by utilizing a general tensor ring (TR) representation with a sequence of latent cores BID21 . One motivation of TR for MTL is it generalizes other chain structured tensor networks , especially tensor train (TT) BID18 , in terms of model expressivity power, as TR can be formulated as a sum of TT networks. On the other hand, TR is able to approximate tensors using lower overall ranks than TT does , thus yielding a more compact and sparselyconnected model with significantly less parameters for deep MTL. Adopting TR-format with much lower ranks could bring more benefits to deep MTL if we tensorize a layer-wise weight of each task into a higher-order weight tensor, since the weight can be decomposed into a relatively larger number but smaller-sized cores. This in turn facilitates the sharing of cores at a finer granularity and further enhances the effectiveness of sharing. Additionally, BID34 In this paper, we have introduced a novel knowledge sharing mechanism for connecting task-specific models in deep MTL, namely TRMTL. The proposed approach models each task separately in the form of TR representation using a sequence latent cores. Next, TRMTL shares the common knowledge by ting any subset of layer-wise TR cores among all tasks, leaving the rest TR cores for private knowledge. TRMTL is highly compact yet super flexible to learn both task-specific and task-invariant features. TRMTL is empirically verified on various datasets and achieves the stateof-the-art results in helping the individual tasks to improve their overall performance. Table 4 : Performance comparison of STL, MRN, DMTRL and our TRMTL on CIFAR-10 with unbalanced training samples, e.g., '5% vs 5% vs 5%' means 5% of training samples are available for the respective task A, task B and task C. TR-ranks R = 10 for TRMTL.
a deep multi-task learning model adapting tensor ring representation A variant of tensor ring formulation for multi-task learning by sharing some of the TT cores for learning "common task" while learning individual TT cores for each separate task