I'm new to machine learning. I have text, and I tag the text according to their parts of speech tag ie walk is tagged as verb, etc. I tag entire sentences, and then convert them into a vector based on their frequency like so:

Dan, walk the dog - > noun,verb, definite-article, noun -><2,1,1> (2x nouns, 1 verb, 1x DA).

The sentences each belong to one of two classes. I've performed a Kolmogorov–Smirnov test on each tag and picked the ones with the lowest P-value to use. I've used a couple of the models provided by scikit but each time I'm only getting around 55-56% accuracy. The models I've tried are: Gaussian Naive Bayes, K-Nearest Neighbours, SVM, Decision Tree, Random Forest Classifier, Quadratic Discriminant Analysis...

I'm asking if anyone has any comments or pointers about these methods? Should I be using different machine learning methods? Am I going about this problem completely wrong? I would really appreciate any input.

  • 1
    $\begingroup$ I don't understand what you're trying to achieve -- can you try clarifying that? 55% accuracy at what? Also, I'm not clear on how you are using the K-S test or what purpose it serves. $\endgroup$ – D.W. Apr 24 '15 at 7:13
  • $\begingroup$ I don't think any of those models will perform well. You're completely disregarding sentence structure and boiling it down to feature counts. I think more classically suited methods such as HMMs, or other more sophisticated NLP-based approaches will trounce these non-structural approaches. $\endgroup$ – Nicholas Mancuso Apr 24 '15 at 22:18
  • $\begingroup$ all fine, but what are the two classes representing? yes there are custom algorithms for language/ text analysis that may do better... also unless you have some indication otherwise, that may be a near-optimal extraction rate given no other info. for example, what is the rate when classified by humans etc? $\endgroup$ – vzn Jun 21 '15 at 1:37

You're trying to classify the sentences' sources, right? I would suggest grouping your training data by their sources (but only up to some fixed size, in which case you would have a duplicate entry for that source). E.g., if you are trying to classify the source of a tweet, you would make each tweet (not sentence, and that means each vector would represent a string of up to 140 characters) a vector and calculate its corresponding features, which should span the contents that the feature vector would relate to. Further if you could get each of the fixed bodies of text into ARFF format, where the last value in the vector would represent the source of the fixed body of text and the feature vectors would each have the same number of features (the features of which you would pick, more on this in a bit) then you wouldn't be too far from being able to use the Weka suite of classifiers. As for which classifier you should use, I suggest skimming through the following paper:

Antti Puurula, Sung-Hyon Myaeng. Integrated Instance- and Class-based Generative Modeling for Text Classification. ADCS 2013.

It comes from the same machine learning group that produced the Weka suite of classifiers and should give you some pretty good insight.

For your choice of what features you would like to vectorize the fixed bodies into, it would be good to count classes of tags. For example, you could count the number of adverbs as a feature, and/or count the number of first-person/second-person/third-person pronouns as a feature, and/or count the number of verbs in a certain tense as a feature, etc. The features you choose are more so based on your understanding of the bodies. If your fixed bodies could include multiple sentences then some other features could also be the average length of sentences (with respect to number of tokens), the average length of tokens (omitting punctuation, for example, from this group), and/or even just the number of sentences in the fixed body.

Make sure to pick enough features, and make sure to pick features you think would be better at differentiating your classes of sources. For example, if any two of your sources are likely to differ in the formality of their tones, then a useful feature would be the number of slang tokens per fixed body.


Your Answer

