Word embeddings are widely used in machine learning based natural language processing systems. It is common to use pre-trained word embeddings which provide benefits such as reduced training time and improved overall performance. There has been a recent interest in applying natural language processing techniques to programming languages. However, none of this recent work uses pre-trained embeddings on code tokens. Using extreme summarization as the downstream task, we show that using pre-trained embeddings on code tokens provides the same benefits as it does to natural languages, achieving: over 1.9x speedup, 5\% improvement in test loss, 4\% improvement in F1 scores, and resistance to over-fitting. We also show that the choice of language used for the embeddings does not have to match that of the task to achieve these benefits and that even embeddings pre-trained on human languages provide these benefits to programming languages.    One of the initial steps in a machine learning natural language processing (NLP) pipeline is converting the one-hot encoded R V tokens into dense R D representations, with V being the size of the vocabulary, D the embedding dimensions and V << D. This conversion is usually done with a single layer neural network, commonly called an embedding layer.The parameters of the embedding layer can either be initialized randomly or initialized via "pretrained" parameters obtained from a model such as word2vec BID22 a) , GloVe BID25 or a language model BID17 BID11 BID26 .It is common to use pre-trained parameters (most frequently the GloVe embeddings), which act as a form of transfer learning BID23 similar to that of using pre-trained parameters for the convolutional kernels in a machine learning computer vision task BID12 BID14 . These parameters in the embedding layer are then fine-tuned whilst training on the desired downstream task.The use of these pre-trained embeddings over random initialization allows machine learning models to: train faster, achieve improved overall performance BID13 , increase the stability of their training, and reduce the amount of over-fitting BID23 .Recently there has been an increased interest in applying NLP techniques to programming languages and software engineering applications BID30 BID3 , the most common of which involves predicting the names of methods or variables using surrounding source code BID27 BID0 BID4 BID6 a) .Remarkably , none of this work takes advantage of pre-trained embeddings created on source code. From the example below in table 1, we can see how semantic knowledge (provided by the pre-trained code embeddings) of the method body would help us predict the method name, i.e. knowing how pi and radius are used to calculate an area and how height and width are used to calculate an aspect ratio.float getSurfaceArea (int radius) { return 4 * Math.PI * radius * radius; } float getAspectRatio (int height, int width) { return height / width; } Table 1 : Examples showing how the semantics of the variable names within a method can be used to reason about the name of the method body This semantic knowledge is available to us as even though computers do not need to understand the semantic meaning of a method or variable name, they are mainly chosen to be understood by other human programmers BID10 .In this paper , we detail experiments using pre-trained code embeddings on the downstream task of predicting a method name from a method body. This task is known as extreme summarization BID2 as a method name can be thought of as a summary of the method body. Our experiments are focused on answering the following research questions:1. Do pre-trained code embeddings reduce training time?2. Do pre-trained code embeddings improve performance?3. Do pre-trained code embeddings increase stability of training?4. Do pre-trained code embeddings reduce over-fitting?5. How does the choice of corpora used for the pre-trained code embeddings affect all of the above?To answer RQ5, we gather a corpus of C, Java and Python code and train embeddings for each corpus separately, as well as comparing them with embeddings trained on natural language. We then test each of these on the same downstream task, extreme summarization, which is in Java. We also release the pre-trained code embeddings . We refer back to our research questions.Do pre-trained code embeddings reduce training time? Yes, tables 3 and 4 show we get an average of 1.93x speedup. This is correlated with the amount of overlap between the task vocabulary and the embedding vocabulary, shown in figure 2a.Do pre-trained code embeddings improve performance? Yes, tables 3 and 4 show we get an average of 5% relative validation loss improvement. Again, this is correlated with the amount of overlap between the vocabularies, shown in figure 2b.Do pre-trained code embeddings increase stability of training? Although this is difficult to quantify due to how over-fitting interacts with the variance of the validation loss curves, from figures 1a and 1c we can see a clear increase in the variance of the validation loss curves using random embeddings compared to those using pre-trained embeddings.Do pre-trained code embeddings reduce over-fitting? Yes, tables 6 and 7 show that the random embeddings over-fit more than the pre-trained embeddings on every project. However, this does not seem to have a correlation with the amount of vocabulary overlap and further work is needed to determine the cause of this.How does the choice of corpora used for the pre-trained code embeddings affect all of the above? Intuitively, it would seem the best pre-trained embeddings would be those that are trained on the same language as that of the downstream task, but this is not the case. We hypothesize through the examples shown in tables 1 and 5 that the differing syntax between the languages is not as important as sensible semantic method and variable names within the dataset. This semantic information is also contained in human languages, which explains why the English embeddings also receive comparable performance.
Researchers exploring natural language processing techniques applied to source code are not using any form of pre-trained embeddings, we show that they should be. This paper sets to understand whether pretraining word embeddings for programming language code by using NLP-like language models has an impact on extreme code summarization task. This work shows how pre-training word vectors using corpuses of code leads to representations that are more suitable than randomly initialized and trained representations for function/method name prediction