Curriculum learning and Self paced learning are popular topics in the machine learning that suggest to put the training samples in order by considering their difficulty levels. Studies in these topics show that starting with a small training set and adding new samples according to difficulty levels improves the learning performance. In this paper we experimented that we can also obtain good results by adding the samples randomly without a meaningful order. We compared our method with classical training, Curriculum learning, Self paced learning and their reverse ordered versions. Results of the statistical tests show that the proposed method is better than classical method and similar with the others. These results point a new training regime that removes the process of difficulty level determination in Curriculum and Self paced learning and as successful as these methods.  BID1 named Curriculum learning the idea of following an order related with difficulty of the samples during training which provides an optimization for non convex objectives. After this many researchers tried to find the most efficient curriculum to get the best yield with this approach. In BID15 's study conventional curriculum learning did not work so well and they developed a new version. BID14 proposed three different curriculum strategies for language model adaptation of recurrent neural networks. In the field of computer vision BID13 looked for the best order of tasks to learn. Although these models have better generalization performance with the proposed curriculum methods it is not known whether the tried methods ensures the best curriculum.A curriculum is work-specific so could not be applicable for another work. In order to use the curriculum logic in different applications BID10 suggested a method that the learner decides itself which samples are easy or difficult at every stage. This method called Self paced learning was combined with Curriculum learning which provides prior information by BID9 . In another work BID7 introduced a method to automatically select the syllabus to follow for the neural networks. BID12 also proposed a way to learn simple subtasks before the complex tasks and achieved better results than using manually designed curriculum.In some cases higher learning performance could be obtainable by adding some noises to easy-tohard ordering of the samples. BID8 gave preference to both easy and diverse samples and outperform the conventional Self paced learning BID10 ) algorithm. Emphasizing the uncertain samples suggested by BID3 lead to more accurate and robust SGD training. BID0 explored the inversed versions of the Self paced learning and Self paced learning with diversity BID8 ) and demonstrated that these methods performed slightly better than their standard variants. Consistent with the literature we have showed in our previous work () that using both curriculum and anti-curriculum strategies improving generalization performance in a wide application area. These researches brings a question to minds: While it is natural and logical to obtain better results by sorting the samples from easy-to-hard why it is also better to sort the samples from hard-to-easy?In this study we point that to start with a small training set and add new samples in both curriculum and anti-curriculum learning makes these methods better. So we claim that it is possible to have better results only by adding new samples stage-by-stage without a meaningful order. We experimented two ordering types related with difficulty (easy-to-hard and hard-to-easy) and our method without a meaningful order. Training was carried out by adding a new group to the training set at every stage. We compared the proposed method with two strategies. First one is Curriculum learning which we give the difficulty levels of the samples as pre-information. Second one is Self paced learning which the trained network determines the difficulty levels of the samples at each stage. All methods including usual baseline training have been compared by using paired T-test and the results are examined. We drew our attention that both versions of training with easy-to-hard ordered and hard-to easy ordered samples have better performance. That led us to investigate what common issues they have. We considered that their common point is growing the training sets during training. Therefore, instead of ordering the samples according to difficulty we only added some samples randomly at each stage. In these experiments we obtained similar results with Curriculum, Anti-curriculum, Self Paced and Self Paced-Inversed methods which are related to difficulty levels. According to these results, we can claim that the success of Curriculum learning and Self paced learning approaches not comes from the fact that they follow a meaningful order but trained by growing training sets.In FIG0 we showed some examples for the individual instances. We started the optimization from θ A , instances under this point are considered as easy and above are difficult. If we take an easy instance it is possible or not to guide the optimization to a better minimum. It will be stop at the local minimum θ B in the worst case. Similarly if we take a difficult instance it is possible or not to obtain a better result. Implementation results also showed that both easy-to-hard and hard-to-easy ordered methods can be successful. Therefore ordering of the samples are not so important to guide the optimization.It is a better situation to shorten the distance between θ B and θ C in FIG2 to bypass the local minimum. When the points are same for a saddle point, training with growing sets will probably overcome this point and find a better minimum. This is a good condition when considering saddle points are so much than local minimums in high dimensional functions as mentioned in BID4 .On many data sets with different distributions we used ensemble method to automatically determine the difficulty of the samples for curriculum learning. Pre-processing for difficulty level determination can be thought to caused slowdown. However it has provided a faster neural network training than SPL. Also it could be said that ensemble method set a better ordering than SPL by considering their number of wins against Baseline.ACL and SPLI, which are the inverse versions of the CL and SPL methods, has performed poorly in some high error rated data sets. The effect of giving the samples at different points during the training has been studied in BID5 . In these methods , noisy examples may be effecting the output more because of giving at the beginning. Nevertheless, the inverse versions of the approaches have better performance than their standard versions. However, CL and SPL methods did not lose in any data set so this shows they have a robust aspect. It is thought that these methods must have a theoretical explanation about ensuring resistance to noises. BID6 studied on why these methods are effectiveness especially on big and noisy data.SPLI method has the most winning against Baseline. In this method strategy of selecting the samples to learn at each stage reminds pool-based active learning BID11 ) in which the learner wants to learn the uncertain samples of the unlabeled data pool. Also non-loss of CL and SPL, and more wins of ACL and SPLI shows the necessity of determining the valuable-example-based curriculum instead of easiness-based-curriculum for the future work.
We propose that training with growing sets stage-by-stage provides an optimization for neural networks. The authors compare curriculum learning to learning in a random order with stages that add a new sample of examples to the previously, randomly constructed set This paper studies the influence of ordering in the Curriculum and Self paced learning, and shows that to some extent the ordering of training instances is not important.