{"action":"opened","issue":{"url":"https://api.github.com/repos/nltk/nltk/issues/1033","labels_url":"https://api.github.com/repos/nltk/nltk/issues/1033/labels{/name}","comments_url":"https://api.github.com/repos/nltk/nltk/issues/1033/comments","events_url":"https://api.github.com/repos/nltk/nltk/issues/1033/events","html_url":"https://github.com/nltk/nltk/issues/1033","id":92824391,"number":1033,"title":"AbstractCollocationFinder from_documents() issues","user":{"login":"fievelk","id":2814672,"avatar_url":"https://avatars.githubusercontent.com/u/2814672?v=3","gravatar_id":"","url":"https://api.github.com/users/fievelk","html_url":"https://github.com/fievelk","followers_url":"https://api.github.com/users/fievelk/followers","following_url":"https://api.github.com/users/fievelk/following{/other_user}","gists_url":"https://api.github.com/users/fievelk/gists{/gist_id}","starred_url":"https://api.github.com/users/fievelk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fievelk/subscriptions","organizations_url":"https://api.github.com/users/fievelk/orgs","repos_url":"https://api.github.com/users/fievelk/repos","events_url":"https://api.github.com/users/fievelk/events{/privacy}","received_events_url":"https://api.github.com/users/fievelk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-07-03T08:40:49Z","updated_at":"2015-07-03T08:40:49Z","closed_at":null,"body":"If we pass a set of documents (e.g. tweets) to the `from_documents()` method we also get bigrams that never appeared in any document. This is due to `_itertools.chain(*documents)`, which collapses all the documents (so we end up with a single big document and we extract *its* bigrams).\r\n\r\n    def from_documents(cls, documents):\r\n        \"\"\"Constructs a collocation finder given a collection of documents,\r\n        each of which is a list (or iterable) of tokens.\r\n        \"\"\"\r\n        return cls.from_words(_itertools.chain(*documents))\r\n\r\nFor example,using a list of two documents:\r\n    \r\n    docs = [['I','am','a','farmer'], ['Newspapers','are','interesting']]\r\n\r\n    finder = BigramCollocationFinder.from_documents(docs)\r\n    finder.nbest(BigramAssocMeasures.raw_freq, 10)\r\n\r\n    >>> [('I', 'am'), ('Newspapers', 'are'), ('a', 'farmer'), ('am', 'a'), ('are', 'interesting'), ('farmer', 'Newspapers')]\r\n\r\nSo the method will also generate the bigram `('farmer', 'Newspapers')`, which actually never occurs in the documents. \r\n\r\nCon we consider it as an issue?"}}