I am dealing with a typical $p > n$ problem in the medical field. (typically $p \approx 3700$ and $n \approx 100$ ). The dependent variable is binary (healthy/sick) and features are continuous variables representing intensities of a large set of bio-markers. The bio-markers (i.e. features) are extracted from samples using a feature selection algorithm developed internally. By the nature of the algorithm, the extracted features are dependent on the data set.

Goal is to extract features which are relevant for the outcome of the conditional variable (healthy/sick).

Process: Since we are dealing with little sample sizes, we suggest to use cross validation for the feature selection, rather than applying the algorithm to the whole set, as follows:

  1. Split original data into testing (10%)/training(90%) data sets.
  2. Split training data set 10 times into 10 folds (CV).
  3. In each step of CV apply feature selection algorithm, followed by a non-parametric U-test and sort by FDR adjusted p-values.
  4. In each step select features with FDR adjusted p-value $\leq 0.05$.
  5. After CV is finished, collect features from every CV step into one final data set and order them by the number appearances over all steps of CV.
  6. Select top $k$ (e.g. $k = 10$) features from the final list.
  7. Develop a classifier (e.g. SVM) by training the selected features from step 6 on the complete training data set.
  8. Test predictive performance of selected features + model by measuring accuracy on the testing data from step 1.


  1. Does the above procedur make sense? Is there any reference in literature for only selecting features using CV (steps 2 - 6)?

  2. Following https://www.sciencedirect.com/science/article/pii/S0933365715001426 we do not want to train any classifier during the steps in CV. However, if we would use the final list of features and fed it back into CV to train a classier at every step, would that not also introduce bias?

Thank you.

Edit In order to avoid lengthy comments below, the question boils down to: Can CV be used purely for feature selection (reduction) as in steps 3-6?

  • 1
    $\begingroup$ Have you tried something like a random forest? It might accomplish a fair amount of what you're after $\endgroup$ – mkt - Reinstate Monica Sep 3 '18 at 14:48
  • $\begingroup$ Good idea and yes we tried :) RF can indeed replace the feature selection algorithm we use. However, our question is more about the actual procedure. Which algo goes into feature selection is indeed a matter of choice. Thx $\endgroup$ – JJR Sep 3 '18 at 14:57
  • 1
    $\begingroup$ The trained RF could also be used as your classifier, simplifying this whole process considerably. I'm not entirely clear on your broader goal, though. Including more information about that might get you better answers. $\endgroup$ – mkt - Reinstate Monica Sep 3 '18 at 15:06
  • 1
    $\begingroup$ The answer might depend on the scale and nature of the data set. The best way to proceed with only a few hundred cases might not be the best way to proceed if you have tens of thousands of cases. The relative prevalence of healthy/sick cases could also be important. Also, please say a bit more about the purposes to which you intend to apply your model (inference, prediction, etc). On this site, it's most useful to add this information by editing the question itself, as comments are sometimes deleted and those who use the site might miss information that's only in comments. $\endgroup$ – EdM Sep 3 '18 at 16:45
  • 2
    $\begingroup$ You just don't have enough data for this to work in your situation. It may help you to read my answer here: Sane stepwise regression? $\endgroup$ – gung - Reinstate Monica Sep 5 '18 at 13:44

Although one might use CV for feature selection as you propose, the proposed approach might not be the wisest way to proceed, particularly with your limited data set.

There's already a problem with Steps 1 and 8. A clean test/training separation (Step 1) can be helpful with very large data sets, but there's a serious problem in this case: your completely held-out test set would only be 10 cases, with at most 5 in the smaller of the healthy/sick classes. That's not going to provide, in Step 8, a very rigorous test of the quality of your feature selection. That problem of limited sample size carries over into the CV proposed in Step 2 and following, where you are now down to only 9 held-out cases in each fold. If your smaller class is only at 20-30% prevalence, you will often have no held-out test members in the smaller class in CV folds, limiting its usefulness.

The second problem is the idea that you can use CV "purely for feature selection" without "measuring performance of the process." The "most significant features" are precisely those that provide "a usable model" for describing the population of interest. That assessment must be based on some measure of how well a model based on the selected features represents the data, even if prediction is not your intended future use of the model. Isn't that what you do in effect for every fold of the proposed CV?

In a fundamental sense, what you validate with CV or almost any statistical tool is the combination of the process and the result, not just the result (in your case, the selected features). For example a p-value in frequentist statistics represents the probability that, if you repeated the experiment (process) a large number of times, you would get a result so extreme as what you observed based simply on chance. And if you want to select 10 features based on your present data sample, wouldn't you want some assurance that the result is likely to be useful with another sample from the same population?

The guiding principle for validating feature selection while minimizing bias is that all steps of the modeling/selection should be repeated for all folds of CV or bootstrapping. This issue is discussed in many threads on this site, with this and this being two threads particularly relevant to your problem.

You could apply this principle to your problem, but with the limited data available a change of perspective might help: accept that some bias is inevitable resulting from feature selection, but do your best to estimate its magnitude. Find the best set of features based on the entire data set and then use CV or, perhaps better, bootstrapping* to estimate generalizability and bias. Your selected features will be based directly on all the information you have available, and you document to your audience how well your overall modeling approach appears to work.

One more warning: with 100 cases you have no more than 50 in the smaller class. A general rule of thumb for classification is that you are in danger of overfitting/bias if you choose more unpenalized predictors than 1/10 to 1/20 of the number of cases in the smaller class. So if you go for any more than about 5 predictors in your case and aren't penalizing them in some way, you may be in danger of overfitting. You might consider the number of features to select as an object for CV, as is done with LASSO.

*Bootstrapping is useful for estimating how well the combination of process and result would generalize to the population as a whole. The principle is that each bootstrapped sample is related to the data sample at hand as the data sample at hand is related to the whole population. From each of many bootstrapped samples, perform your feature selection algorithm and evaluate its performance on the full data sample in terms of accuracy, bias, or whatever quality measures you use. The pooled quality estimate among the multiple models from bootstrapping against the data sample thus is an estimate of how well your feature selection based on the data sample represents the entire population.

  • $\begingroup$ Thank you for very interesting insights - it's a nice read. Particularly, I found very interesting how you treat feature selection as a process, because I always looked at the combination of feature selection + model development as a process. One thing I don't understand is: Do you mean that taking the union of all features selected from each CV step presents a bias? I ask this, since we typically get different feature set in every CV step. $\endgroup$ – JJR Sep 4 '18 at 7:32
  • $\begingroup$ @JJR I don't know if taking the union of all features selected from each CV would lead to bias, but the approach I suggest could estimate such bias. Insofar as your feature-selection process might favor particular combinations of features in each fold, there's a danger that the union of features from multiple CV folds without attention to co-selection within folds might tend to wash out useful information. That (admittedly non-quantified) fear is one reason to do feature selection on the entire data set and then use bootstrapping to estimate the actual (and probably unavoidable) bias. $\endgroup$ – EdM Sep 4 '18 at 21:53
  • $\begingroup$ Following your elaboration I am actually thinking of doing 10 times 10-fold CV on the whole data set to measure the performance of the complete process: pre-processing + feature selection + supervised model. If the mean accuracy is high enough, then applying the same thing to the whole data set, without any testing data set. As for testing, either refer to a later study with bigger samples or maybe bootstrap (which I personally do not prefer) $\endgroup$ – JJR Sep 5 '18 at 15:10
  • $\begingroup$ That seems like a good way to go, probably much more reliable than the leave-one-out CV that seems to be at the basis of the paper you cite in the question (and that let them avoid the variability in selected features among folds that you found). You might want to look at this thread for discussion of CV versus bootstrapping. Do note that you will best be served by using a proper scoring rule for evaluation, as Frank Harrell points out in his answer on the page linked in this comment and as @cbeleites notes in his answer to your question. $\endgroup$ – EdM Sep 5 '18 at 15:36

Some points in addition to @EdM's nice answer.

  • To give you a feeling for the small sample size trouble you're in (and I thoroughly know very similar situations) have a look at confidence intervals for estimating a proportion (such as accuracy, sensitivity, specificity, predictive values - you name it): Brown, L.; Cai, T. & DasGupta, A.: Interval Estimation for a Binomial Proportion Statistical Science, 2001, 16, 101-133 may be a good starting point.
    In R, binom::binom.confint() calculates confidence intervals using a variety of approximations. If just 1 of the 10 patients reserved in the hold-out step is misclassified, the point estimate is 90 % correct, but the 95 % confidence interval ranges all the way from a bit below 60 % to 100 %, with 2 misclassified it would cover everything between guessing (50 %) and almost perfect (95 %).
    My pragmatic point of view from situations with n and p similar to yours is: the results for the various approximations may look quite different, but these differences IMHO occur in situations where the interpretation in any case is that the model and/or the data set* is unusable in practice. Only, I stay clear of the normal approximation - our situations are clearly outside the recommended conditions where it can be used.
    * Sometimes it is useful to make a back-of-the-envelope calculation before starting to dig into the data: if 100 % accuracy would have a confidence interval that is too wide to be useful, back to the lab is the only possible answer.

  • Conclusion from that is that your verfication/validation of the final model really should use a resampling strategy (cross validation or out-of-bootstrap). (see here for considerations when independent test sets may be a better choice - as opposed to a hold-out that was split from the data just as a cross validation splitting)

  • I refrain from data/validation-driven optimization in such situations: these optimizations crucially need to be able to assess performance with a precision that allows to distinguish between the best and good models. No way to get that here.

  • The situation is somewhat better if you look at so-called proper scoring rules. But they don't work miracles, neither.

  • Still, I agree with EdM to go for a LASSO, random forest, or any other type of model and feature selection where you can use external knowledge for guidance.

  • In the end, it may be better to fix the desired complexity (e.g. number of features to be kept) beforehand: that will leave you with a model and its validation results. Not necessarily the optimal model - but data-driven optimization is at least equally unlikely to achieve this in the present situation.

  • There is no universal best set of features. There may be a best set of features to train that classifier: good feature selection needs to take into account data (data generating processes) as well as the needs/strengths/weaknesses of the classifier to be trained. It really is part of the training.

  • $\begingroup$ +1 Thank you for the great addon, especially your elaboration on the confidence intervals is very nice. Note that we are completely aware of the dimensionality problem. We did not only tried Wilcoxon test as a feature selection method, but also Lasso, Ridge and elastic net. My question was more about the procedure and not about a specific feature selection method. Anyway, your answer enriched my thought process significantly :) As such I am thinking of the following process: pre-processing + feature selection with Wilcox q-value $\leq 0.05$ + supervised ML (e.g. RF or SVM for top 5). $\endgroup$ – JJR Sep 5 '18 at 15:02

Your Answer

