Take the 2-minute tour ×

I have a crawler program that stores sport data from 7 difference news agencies every day. it stores about 1200 sport news every day. I want to categorize news of last two days into sub-categories. So every two days I have about 2400 news that are exactly for these days and many of their topics are talking exactly about the same happening. for example:

70 news are talking about 500 miles racing of Brad Keselowski.

120 news are talking about US swimmer Nyad that begins swimming.

28 new are talking about the match between Man United and Man City.

. . .

In other words, I want to make something like Google News.

The problem is that this situation is not a classification problem, because I don't have special classes. for example, my classes are not swimming, golf, football, etc. my classes are a special events in every field that happened in these two years. So I cannot use classification algorithms such as Naive Bayes.

On the other hand, my problem is not solving with clustering algorithms too. Because I don't want to force them to put into n clusters. Maybe one of the news doesn't have any similar news or maybe in one pack of two days, there are 12 different stories, but in other two days, there are 30 different issues. So I cannot use clustering algorithms such as "Single Link( Maximum Similarity)", "Complete Link( Minimum Similarity)", "Maximum Weighted Matching" or "Group Average( Average Intra Similarity)".

I have some ideas myself to do this, for example, each two news that have 10 common words, should be in the same class. But if we don't consider some parameters such as length of documents, influence of common and rare words and some other things, this will not work well.

I have read this paper, but it was not my answer.

Is there any known algorithm to solve this problem?

share|improve this question
I think you are definitely looking for clustering algorithms here. Clustering algorithms exist that don't always produce some fixed number of clusters but instead try to find groups of elements that look similar. Don't dismiss this as a possibility; a huge amount of AI research goes into this! –  templatetypedef Aug 8 '11 at 8:41

4 Answers 4

The problem strikes me as a clustering problem with an unknown quality measure for the clusters. That points to an unsupervised method, which is ultimately based on detecting correlations using redundancy in the data. Perhaps something like principal component analysis or latent semantic analysis could be useful. The different dimensions (principal components or singular vectors) would indicate distinct major themes, with the terms corresponding to the vector components hopefully being the words appearing in the description. One drawback is that there's no guarantee that the strongest correlations would lead easily to a sensible description.

share|improve this answer

You might use hierarchical clustering algorithms to investigate relationships between your items - the closest items (news with almost the same description) would be in the same clusters, and the closest clusters (groups of similar news) would be in the same super-cluster etc. Also, there is pretty nice and fast algorithm called CLOPE - http://www.google.com.ua/url?sa=t&source=web&cd=11&sqi=2&ved=0CF0QFjAK&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.

share|improve this answer

Take a look at "topic models" and "Latent Dirichlet Allocation". These are popular and you'll find code in a variety of languages.

share|improve this answer

There are many document clustering algorithms out there. Take a look at "Hierarchical document clustering using frequent itemsets", for example, and see if that is similar to what you want. If you're programming in Java, you may get some mileage out of the S-space package, which includes algorithms for latent semantic analysis (LSA) among others.

share|improve this answer

Your Answer


