Take the 2-minute tour ×

I want to double check some concepts I am uncertain of regarding the training set for classifier learning. When we select records for our training data, do we select an equal number of records per class, summing to N or should it be randomly picking N number of records (regardless of class)?

Intuitively I was thinking of the former but thought of the prior class probabilities would then be equal and not be really helpful?

share|improve this question

3 Answers 3

It depends on the distribution of your classes and the determination can only be made with domain knowledge of problem at hand. You can ask the following questions:

  • Are there any two classes that are very similar and does the learner have enough information to distinguish between them?
  • Is there a large difference in the prior probabilities of each class?

If so, you should probably redistribute the classes.

In my experience, there is no harm in redistributing the classes, but it's not always necessary.

It really depends on the distribution of your classes. In the case of fraud or intrusion detection, the distribution of the prediction class can be less than 1%. In this case you must distribute the classes evenly in the training set if you want the classifier to learn differences between each class. Otherwise, it will produce a classifier that correctly classifies over 99% of the cases without ever correctly identifying a fraud case, which is the whole point of creating a classifier to begin with.

Once you have a set of evenly distributed classes you can use any technique, such as k-fold, to perform the actual training.

Another example where class distributions need to be adjusted, but not necessarily in an equal number of records for each, is the case of determining upper-case letters of the alphabet from their shapes.

If you take a distribution of letters commonly used in the English language to train the classifier, there will be almost no cases, if any, of the letter Q. On the other hand, the letter O is very common. If you don't redistribute the classes to allow for the same number of Q's and O's, the classifier doesn't have enough information to ever distinguish a Q. You need to feed it enough information (i.e. more Qs) so it can determine that Q and O are indeed different letters.

share|improve this answer
i read about k-fold, as provided by @fyr. I'm confused. After I have the evenly distributed training sets per class, which sets do i provided to train the classifier for actual classifying use? My prior is i believe even: like c1: 90% and c2: 10%. –  goh Jul 6 '11 at 7:07
You create a whole new data set with evenly distributed classes. You then use this new data and partition it into training, cross-val, and test as needed. k-fold is a very common way to partition the data into training and cross-val. –  dacamo76 Jul 6 '11 at 15:37

The preferred approach is to use K-Fold Cross validation for picking up learning and testing data.

Quote from wikipedia:

K-fold cross-validation



You should always take the common approach in order to have comparable results with other scientific data.

share|improve this answer

I built an implementation of a Bayesian classifier to determine if a sample is NSFW (Not safe for work) by examining the occurrence of words in examples. When training a classifier for NSFW detection I've tried making it so that each class in the training sets has the same number of examples. This didn't work out as well as I had planned being that one of the classes had many more words per example than the other class.

Since I was computing the likelihood of NSFW based on these words I found that balancing out the classes based on their actual size (in MB) worked. I tried 10-cross fold validation for both approaches (balancing by number of examples and size of classes) and found that balancing by the size of the data worked well.

share|improve this answer

Your Answer


