An Experiment on Feature Selection using Logistic Regression
Raisa Islam Subhasish Mazumdar Rakibul Islam
In supervised machine learning, feature selection plays a very important role
by potentially enhancing explainability and performance as measured by
computing time and accuracy-related metrics. In this paper, we investigate a
method for feature selection based on the well-known L1 and L2 regularization
strategies associated with logistic regression (LR). It is well known that the
learned coefficients, which serve as weights, can be used to rank the features.
Our approach is to synthesize the findings of L1 and L2 regularization. For our
experiment, we chose the CIC-IDS2018 dataset owing partly to its size and also
to the existence of two problematic classes that are hard to separate. We
report first with the exclusion of one of them and then with its inclusion. We
ranked features first with L1 and then with L2, and then compared logistic
regression with L1 (LR+L1) against that with L2 (LR+L2) by varying the sizes of
the feature sets for each of the two rankings. We found no significant
difference in accuracy between the two methods once the feature set is
selected. We chose a synthesis, i.e., only those features that were present in
both the sets obtained from L1 and that from L2, and experimented with it on
more complex models like Decision Tree and Random Forest and observed that the
accuracy was very close in spite of the small size of the feature set.
Additionally, we also report on the standard metrics: accuracy, precision,
recall, and f1-score.