{"action":"created","issue":{"url":"https://api.github.com/repos/nltk/nltk/issues/978","labels_url":"https://api.github.com/repos/nltk/nltk/issues/978/labels{/name}","comments_url":"https://api.github.com/repos/nltk/nltk/issues/978/comments","events_url":"https://api.github.com/repos/nltk/nltk/issues/978/events","html_url":"https://github.com/nltk/nltk/issues/978","id":76290341,"number":978,"title":"formula TF is not canonical","user":{"login":"ermakovpetr","id":454655,"avatar_url":"https://avatars.githubusercontent.com/u/454655?v=3","gravatar_id":"","url":"https://api.github.com/users/ermakovpetr","html_url":"https://github.com/ermakovpetr","followers_url":"https://api.github.com/users/ermakovpetr/followers","following_url":"https://api.github.com/users/ermakovpetr/following{/other_user}","gists_url":"https://api.github.com/users/ermakovpetr/gists{/gist_id}","starred_url":"https://api.github.com/users/ermakovpetr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ermakovpetr/subscriptions","organizations_url":"https://api.github.com/users/ermakovpetr/orgs","repos_url":"https://api.github.com/users/ermakovpetr/repos","events_url":"https://api.github.com/users/ermakovpetr/events{/privacy}","received_events_url":"https://api.github.com/users/ermakovpetr/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-05-14T08:57:18Z","updated_at":"2015-05-15T03:19:47Z","closed_at":null,"body":"nltk/nltk/text.py\r\n\r\n```python\r\ndef tf(self, term, text, method=None):\r\n    \"\"\" The frequency of the term in text. \"\"\"\r\n    return text.count(term) / len(text)\r\n```\r\n\r\nhttp://en.wikipedia.org/wiki/Tf–idf\r\nReal TF is only the number of occurrences of words in text\r\n\r\nAnd there is also a variation of the \"normalization\":\r\n1. Boolean \"frequencies\": tf(t,d) = 1 if t occurs in d and 0 otherwise;\r\n2. Logarithmically scaled frequency: tf(t,d) = 1 + log f(t,d), or zero if f(t, d) is zero;\r\n3. Augmented frequency, to prevent a bias towards longer documents, e.g. raw frequency divided by the maximum raw frequency of any term in the document:\r\n![5cc18acd4dbd9be636047fc2a7a10105](https://cloud.githubusercontent.com/assets/454655/7628683/f5e1a168-fa2e-11e4-9ddd-5791855edcec.png)\r\n4. Normalization of the number of all words\r\n![afcade8c6e040203a292c8ca784fa3f3](https://cloud.githubusercontent.com/assets/454655/7628719/5597925c-fa2f-11e4-812c-c7aeb6d4e48d.png)\r\n\r\nSource: wiki, Introduction to information retrieval, ...\r\n\r\nBut before that code, I have not seen anywhere else the normalization of the length of the text."},"comment":{"url":"https://api.github.com/repos/nltk/nltk/issues/comments/102241738","html_url":"https://github.com/nltk/nltk/issues/978#issuecomment-102241738","issue_url":"https://api.github.com/repos/nltk/nltk/issues/978","id":102241738,"user":{"login":"serapio","id":3781280,"avatar_url":"https://avatars.githubusercontent.com/u/3781280?v=3","gravatar_id":"","url":"https://api.github.com/users/serapio","html_url":"https://github.com/serapio","followers_url":"https://api.github.com/users/serapio/followers","following_url":"https://api.github.com/users/serapio/following{/other_user}","gists_url":"https://api.github.com/users/serapio/gists{/gist_id}","starred_url":"https://api.github.com/users/serapio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/serapio/subscriptions","organizations_url":"https://api.github.com/users/serapio/orgs","repos_url":"https://api.github.com/users/serapio/repos","events_url":"https://api.github.com/users/serapio/events{/privacy}","received_events_url":"https://api.github.com/users/serapio/received_events","type":"User","site_admin":false},"created_at":"2015-05-15T03:19:47Z","updated_at":"2015-05-15T03:19:47Z","body":"Length normalization is included in Lucene's tf-idf, as well as in\nalternative relevance measures like BM25. That said, neither of those\nlength normalizations are linear with the length itself. Lucene's tf-idf\nuses 1/sqrt(len(text)), and in BM25 it is 1/(constants + len(text)).\n\nOn Thu, May 14, 2015 at 8:05 PM, jnothman <notifications@github.com> wrote:\n\n> I agree that this is non-standard, although I have heard of it elsewhere\n> (no citation). Also, something akin to length normalisation occurs with the\n> l2-normalisation of cosine similarity, although that would take the idf\n> into account.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/nltk/nltk/issues/978#issuecomment-102239131>.\n>\n\n\n\n-- \nLucien S. Carroll\nGraduate Student\nUCSD Linguistics\nhttp://ling.ucsd.edu/~lucien\n"}}