Suppose you have a dataset with the following properties:

  1. The number of samples is fairly large (~100K samples)
  2. There are ~150 contextual features and 1 feature consisting of a text-string (which can, of course, be split into any number of features depending on the pre-processing of the text). It is expected that the text-string will have really great predictive power
  3. Samples are divided into 3 categories (prior to you receiving the data) based on a few of the contextual features with category A containing ~5% of the samples, category B containing ~20%, and category C containing the remaining 75%
  4. Category A is entirely labeled, category B is partly labeled (with only a small proportion being unlabeled), and category C is entirely unlabeled
  5. The features used to categorize the samples are likely to influence the probability of a sample belonging to class 0 or class 1.
  6. The samples are not completely different between categories (that is to say, we're not talking cats versus dogs). E.g.: Two very similar samples might end up in different categories based on very small differences on a numerical feature with a large range

The purpose is to build a classifier which will correctly classify the samples. That might look like a semi-supervised learning-problem, but I am worried about the structural differences between the categories. Hence my question: Which strategies could be employed to build a classifier performing well on all of the samples?

Of course I could just be conservative and only deal with the labeled data, but there is great value in also being able to predict the unlabeled data (e.g. the 75% of the data in category C). That's why I'll try to pick your brains for creative solutions!

  • 1
    $\begingroup$ Have you considered clustering ? One of other approach I can think of is to train the model on the labeled data and new incoming data (class C) should be as anomaly ... without sample of data can’t think of of a POC other than the approach used in Cybersecurity when training on “true” data and than predicting the “anomaly”. $\endgroup$
    – n1tk
    Apr 5 '19 at 4:10
  • $\begingroup$ @n1tk Would you mind elaborating on your thoughts on using clustering? I'm struggling to find a way to actually build (and evaluate) a classifier using clustering. And yeah, your other ideas about the anomaly-approach is pretty much what I might end up doing if nothing better pops up. $\endgroup$
    – Mathias
    Apr 6 '19 at 10:16
  • 1
    $\begingroup$ What do you mean by unlabeled? It is not already labeled as category C? If so, what is the problem? Category? Could you provide us with data examples (even if they are fake but hold the same structure as your data) $\endgroup$ Apr 9 '19 at 22:43
  • $\begingroup$ @PedroHenriqueMonforte Sorry for not making myself clear. The task is to predict class 0 or class 1. All of the data is categorized into A, B, and C-classes, but the problem is that only part of the data is labeled in terms of class 0 or class 1. $\endgroup$
    – Mathias
    Apr 11 '19 at 7:10
  • $\begingroup$ You may consider k-NN clustering/classification where you train a model on the samples with labels and then predict labels on the unlabeled data. $\endgroup$
    – Jirka B.
    Apr 23 '19 at 21:28

There is no straightforward way to deal with this scenario, but here are some ideas:

  • Augment the A/B/C label set with an extra label "U" for "unknown". Now all samples are labeled.

  • Predict A/B/C labels from the unlabeled samples from the other pieces of data in the sample. Now all samples are labeled.

  • Train a system for data with labels and another system for data without them. Use them according to the availability of the label. It would be great if your ML approach allows for sharing parts of the model for labeled data and the one for unlabeled data (e.g. neural networks).


Your Answer

