Over the past few years, various tasks involving videos such as classification, description, summarization and question answering have received a lot of attention. Current models for these tasks compute an encoding of the video by treating it as a sequence of images and going over every image in the sequence, which becomes computationally expensive for longer videos. In this paper, we focus on the task of video classification and aim to reduce the computational cost by using the idea of distillation. Specifically, we propose a Teacher-Student network wherein the teacher looks at all the frames in the video but the student looks at only a small fraction of the frames in the video. The idea is to then train the student to minimize  (i)  the difference between the final representation computed by the student and the teacher and/or (ii) the difference between the distributions predicted by the teacher and the student. This smaller student network which involves fewer computations but still learns to mimic the teacher can then be employed at inference time for video classification. We experiment with the YouTube-8M dataset and show  that the proposed student network can reduce the inference time by upto 30% with a negligent drop in the performance. Today video content has become extremely prevalent on the internet influencing all aspects of our life such as education, entertainment, communication etc. This has led to an increasing interest in automatic video processing with the aim of identifying activities BID16 BID24 , generating textual descriptions BID5 , generating summaries BID25 BID13 , answering questions BID8 and so on. On one hand, with the availability of large scale datasets BID18 BID19 BID9 BID0 BID23 for various video processing tasks, it has now become possible to train increasingly complex models which have high memory and computational needs but on the other hand there is a demand for running these models on low power devices such as mobile phones and tablets with stringent constraints on latency and computational cost. It is important to balance the two and design models which can learn from large amounts of data but still be computationally cheap at inference time.With the above motivation, we focus on the task of video classification BID0 and aim to reduce the computational cost at inference time. Current state of the art models for video classification (Yue-Hei BID24 BID10 BID17 treat the video as a sequence of images (or frames) and compute a representation of the video by using a Recurrent Neural Network (RNN). The input to the RNN at every time step is an encoding of the corresponding image (frame) at that time step as obtained from a Convolutional Neural Network. Computing such a representation for longer videos can be computationally very expensive as it requires running the RNN for many time steps. Further, for every time step the corresponding frame from the video needs to pass through a convolutional neural network to get its representation. Such computations are still feasible on a GPU but become infeasible on low end devices which have power, memory and computational constraints.Typically, one can afford more computational resources at training time but a less expensive model is desired at test time. We propose to achieve this by using the idea of distillation wherein we train a computationally expensive teacher network which computes a representation for the video by processing all frames in the video. We then train a relatively inexpensive student network whose objective is to process only a few frames of the video and produce a representation which is very similar to the representation computed by the teacher. This is achieved by minimizing (i) the squared error loss between the representations of the student network and the teacher network and/or (ii) by minimizing the difference between the output distributions (class probabilities) predicted by the two networks. We refer to this as the matching loss. FIG2 illustrates this idea where the teacher sees every frame of the video but the student sees fewer frames, i.e., every j-th frame of the video. At inference time, we then use the student network for classification thereby reducing the time required for processing the video.We experiment with two different methods of training the Teacher-Student network. In the first method (which we call Serial Training), the teacher is trained independently and then the student is trained to match the teacher with or without an appropriate regularizer to account for the classification loss. In the second method (which we call Parallel Training), the teacher and student are trained jointly using the classification loss as well as the matching loss. We experiment with the YouTube-8M dataset and show that the smaller student network reduces the inference time by upto 30% while still achieving a classification performance which is very close to that of the expensive teacher network. The results of our experiments are summarized mainly in Tables 1 (performance) and 4 (computation time). We also report some additional results in Table 2 equally spaced k frames performs better than all the other baselines. The performance gap between Uniform-k and the other baselines is even more significant when the value of k is small. The main purpose of this experiment was to decide the right way of selecting frames for the student network.Based on these results, we ensured that for all our experiments, we fed equally spaced k frames to the student. Also, these experiments suggest that Uniform-k is a strong baseline to compare against.2. Comparing Teacher-Student Network with Uniform-k Baseline: As mentioned above, the Uniform-k is a simple but effective way of reducing the number of frames to be processed. We observe that all the teacher-student models outperform this strong baseline. Further, in a separate experiment as reported in Table 3 we observe that when we reduce the number of training examples seen by the teacher and the student, then the performance of the Uniform-k baseline drops and is much lower than that of the corresponding teacher student network. This suggests that the teacher student network can be even more useful when the amount of training data is limited.3. Serial Versus Parallel Training of Teacher-Student: While the best results in Table 1 are obtained using Serial training, if we compare the corresponding rows of Serial and Parallel training we observe that there is not much difference between the two. We found this to be surprising and investigated this further. In particular, we compared the performance of the teacher after different epochs in the Parallel training setup with the performance of the a static teacher trained independently (Serial). We plotted this performance in FIG3 and observed that after 3-4 epochs of training, the Parallel teacher is able to perform at par with Serial teacher (the constant blue line). As a result, the Parallel student now learns from this trained teacher for a few more epochs and is almost able to match the performance of the Serial student. This trend is same across the different combinations of loss functions that we used. We proposed a method to reduce the computation time for video classification using the idea of distillation. Specifically, we first train a teacher network which computes a representation of the video using all the frames in the video. We then train a student network which only processes k frames of the video. We use different combinations of loss functions which ensures that (i) the final representation produced by the student is the same as that produced by the teacher and (ii) the output probability distributions produced by the student are similar to those produced by the teacher. We compare the proposed models with a strong baseline and skyline and show that the proposed model outperforms the baseline and gives a significant reduction in terms of computational time and cost when compared to the skyline. In particular, We evaluate our model on the YouTube-8M dataset and show that the computationally less expensive student network can reduce the computation time by upto 30% while giving similar performance as the teacher network.As future work, we would like to evaluate our model on other video processing tasks such as summarization, question answering and captioning. We would also like to experiment with more complex and different teacher networks other than the hierarchical RNN considered in this work. We would also like to independently train an agent which learns to select the most favorable k frames of the video as opposed to simply using equally spaced k frames.
Teacher-Student framework for efficient video classification using fewer frames  The paper proposes an idea to distill from a full video classification model a small model that only receives smaller number of frames. The authors present a teacher-student network to solve video classification problem, proposing serial and parallel training algorithms aimed at reducing computational costs.