Learning with a primary objective, such as softmax cross entropy for classification and sequence generation, has been the norm for training deep neural networks for years. Although being a widely-adopted approach, using cross entropy as the primary objective exploits mostly the information from the ground-truth class for maximizing data likelihood, and largely ignores information from the complement (incorrect) classes. We argue that, in addition to the primary objective, training also using a complement objective that leverages information from the complement classes can be effective in improving model performance. This motivates us to study a new training paradigm that maximizes the likelihood of the ground-truth class while neutralizing the probabilities of the complement classes. We conduct extensive experiments on multiple tasks ranging from computer vision to natural language understanding. The experimental results confirm that, compared to the conventional training with just one primary objective, training also with the complement objective further improves the performance of the state-of-the-art models across all tasks. In addition to the accuracy improvement, we also show that models trained with both primary and complement objectives are more robust to single-step adversarial attacks.
 Statistical learning algorithms work by optimizing towards a training objective. A dominant principle for training is to optimize likelihood BID16 , which measures the probability of data given the model under a specific set of parameters. The popularity of deep neural networks has given rise to the use of cross entropy BID13 as its primary training objective, since minimizing cross entropy is essentially equivalent to maximizing likelihood for disjoint classes. Cross entropy has become the standard training objective for many tasks including classification BID12 and sequence generation .Let y i ∈ {0, 1} K be the label of the i th sample in one-hot encoded representation andŷ i ∈ [0, 1] K be the predicted probabilities, the cross entropy H(y,ŷ) is defined as: DISPLAYFORM0 whereŷ ig represents the predicted probability of the ground-truth class for the i th sample. Training with cross entropy as the primary objective aims at findingθ = arg min θ H(y,ŷ), wherê y = h θ (x), h θ is a neural network and x is a sample. Although training using the cross entropy as The model is ResNet-110, and the "embedding" is the vector representation before taking the softmax operation. The embedding representation of each sample is projected to two dimensions using t-SNE for visualization purpose. Compared to ( a), the cluster of each class in (b) is "narrower " in terms of intra-cluster distance. Also, the clusters in (b) seem to have clean and separable boundaries, leading to more accurate and robust classification results.the primary objective has achieved tremendous success, we have observed one limitation: it exploits mostly the information from the ground-truth class as Eq(1) shows; the information from complement classes (i.e., incorrect classes) has been largely ignored, since the predicted probabilities other thanŷ ig are zeroed out due to the dot product calculation with the one-hot encoded y i . Therefore, for classes other than the ground truth, the model behavior is not explicitly optimized -their predicted probabilities are indirectly minimized whenŷ ig is maximized since the probabilities sum up to 1. One way to utilize the information from the complement classes is to neutralize their predicted probabilities. To this end, we propose Complement Objective Training (COT), a new training paradigm that achieves this optimization goal without compromising the model's primary objective. FIG1 illustrates the comparison between FIG1 : the predicted probabilityŷ from the model trained with just cross entropy as the primary objective, and FIG1 :ŷ from the model trained with both primary and complement objectives. Training with the complement objective finds the parameters θ that evenly suppress complement classes without compromising the primary objective (i.e., maximizingŷ g ), making the model more confident of the ground-truth class. Complement objective training requires a function that complements the primary objective. In this paper, we propose "complement entropy" (defined in Section 2) to complement the softmax cross entropy for neutralizing the effects of complement classes. The neural net parameters θ are then updated by alternating iteratively between (a) minimizing cross entropy to increaseŷ g , and (b) maximizing complement entropy to neutralizeŷ j =g . Experimental results (in Section 3) confirm that COT improves the accuracies of the state-of-the-art methods for both (a) the image classification tasks on ImageNet-2012 , Tiny ImageNet, CIFAR-10, CIFAR-100, and SVHN, and (b) language understanding tasks on machine translation and speech recognition. Furthermore, experimental results also show that models trained by COT are more robust to adversarial attacks. In this paper, we study Complement Objective Training (COT), a new training paradigm that optimizes the complement objective in addition to the primary objective. We propose complement entropy as the complement objective for neutralizing the effects of complement (incorrect) classes.Models trained using COT demonstrate superior performance compared to the baseline models. We also find that COT makes the models robust to single-step adversarial attacks.COT can be extended in several ways: first, in this paper, the complement objective is chosen to be the complement entropy. Non-entropy-based complement objectives should also be considered for future studies, which is left as a straight-line future work. Secondly, the exploration of COT on broader applications remains as an open research question. One example would be applying COT on generative models such as Generative Adversarial Networks . Another example would be using COT on object detection and segmentation. Finally, in this work, we show using complement objective help defend single-step adversarial attacks; the behavior of COT on more advanced adversarial attacks deserves further investigation and is left as another future work.A ITERATIVE FAST GRADIENT SIGN METHOD TAB9 shows the performance of the models on the CIFAR-10 dataset under I-FGSM transfer attacks. Generally, the models trained using COT have lower classification error under I-FGSM transfer attacks. The number of iteration is set to 10 in the experiment.
We propose Complement Objective Training (COT), a new training paradigm that optimizes both the primary and complement objectives for effectively learning the parameters of neural networks. Considers augmenting the cross-entropy objective with "complement" objective maximization, which aims at neutralizing the predicted probabilities of classes other than the ground truth labels. The authors propose a secondary objective for softmax minimization based on evaluating the information gathered from the incorrect classes, leading to a new training approach. Deals with the training of neural networks for classification or sequence generation tasks using across-entropy loss