I am trying to get the topic of a sentence, i.e what a sentence is talking about (not the grammatical subject which may be different).

So far I have got:

  1. OpenNLP in Java which is giving me sentence detection, POS tagging, parsing, tokenizer and Name Finder.
  2. MatlParser,stanford Parser - which can give the grammatical subject of a simple sentence by dependency parsing.

I think a noun or a noun phrase will always be the topic in a more general sense, but a sentence can have many nouns and noun phrases.

A complex or compound sentence may have more than one subject which could be considered as the topic of sentence in that case I want to get any one of them. I am looking to get the answer upto some accuracy.

  • Topics are often implicit. And there is really no way to determine the various elements of a sentence with high topicality automatically, I'm afraid. This will only be possible once computer fully understand human language, which means they will have to understand about as much of the world as we do. That will take a while. – Cerberus Oct 4 '12 at 17:25
  • 1
    "I think a noun or a noun phrase will always be the topic". That's not my understanding. In (who went?) "I went", I'd say "went", a verb phrase, is the topic. In "Yesterday, I went", I'd say "yesterday", an adverbial phrase, is the topic. – dainichi Oct 5 '12 at 5:20
  • Yes noun phrase will not be the topic always. Actually i only want to get what peoples are talking about a product and its features in a sentence,so i thought that information will be in noun phrase. – Mahender Singh Oct 12 '12 at 5:46
  • Are you asking just about English or all languages? Be aware that there are languages which distinguish topic and subject grammatically. – hippietrail Oct 31 '12 at 12:20
  • www.uclassify.com - This can also be used for a more general topic classification. This is brute but it can still work in certain scenarios. – arjun Feb 4 '17 at 9:16

While I agree with the comment to your post that this cannot be done perfectly (or even near perfectly) unless computers get a lot better at "understanding" meaning, I'd still say there are a number of heuristics you'd be able to use. Of course, putting them together in a way that it yields useful results would mean a lot of trial and error; actually it might be simpler to use machine learning techniques than to try to hand-craft specific rules. Also consider that you have to be explicit about what you mean by Topic. It is one of those terms that has been used so often, by so many different scholars, that you cannot be sure what it actually means unless you try to provide a precise definition.

That said, here are some patterns I would try to look for:

  • Pronouns tend to be topics more often than full noun phrases, determinate noun phrases more often than indeterminate ones.
  • Topics are often high on the animacy hierarchy.
  • Sometimes, special constructions are used to highlight the topic of a sentence. In English, you can use topicalization where you just put the topical element at the front of the clause (which becomes apparent, when it is not the subject - e.g. "Him I saw yesterday"), but there are also other strategies, like using the "as for" construction.
  • Topics are often continued across sentences, and when the topic is switched we also resort to a number of strategies to make that clear.
  • and so on ... a lot of it will also depend on the language. For example, in some languages (like Japanese or Korean) you have special topic markers
  • Japanese and Korean have both topic markers and subject markers and both languages drop pronouns more often than including them. – hippietrail Oct 31 '12 at 12:21

If you're trying to extract the topic of the text using the words of the text, there are some situations which you may encounter:

1 - You have many samples of the same context (e.g. 20newseltter corpora) which many documents in different topics, and each topic labeled to the topic. In this case you just need to do a simple classification on the new data in the same context i.e. you jave 20,000 documents in 20 topics, you give all these documents to a classifier (e.g. Naive Bayes Classifier) and the new document will be classified easily.

2 - If you don't have any training data, you should use parsers (as I see you're trying the same), in this case you should build PCFGs from what you get from the parsers (POS taggers, name entity recognizers) return to you. This should be made on your type of data specifically and no general rule exists.

3 - If there is a large number of data available but they aren't classified, another way is to use clustering. In many cases in NLP clustering doesn't do much to you but it is worth trying. The only thing you need to make clear for the system is the number of topics (clusters).

Your Answer

