{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4391","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4391/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4391/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4391/events","html_url":"https://github.com/scikit-learn/scikit-learn/issues/4391","id":61250269,"number":4391,"title":"Issue w/ tf-idf computation","user":{"login":"hannawallach","id":873454,"avatar_url":"https://avatars.githubusercontent.com/u/873454?v=3","gravatar_id":"","url":"https://api.github.com/users/hannawallach","html_url":"https://github.com/hannawallach","followers_url":"https://api.github.com/users/hannawallach/followers","following_url":"https://api.github.com/users/hannawallach/following{/other_user}","gists_url":"https://api.github.com/users/hannawallach/gists{/gist_id}","starred_url":"https://api.github.com/users/hannawallach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hannawallach/subscriptions","organizations_url":"https://api.github.com/users/hannawallach/orgs","repos_url":"https://api.github.com/users/hannawallach/repos","events_url":"https://api.github.com/users/hannawallach/events{/privacy}","received_events_url":"https://api.github.com/users/hannawallach/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-03-13T21:25:31Z","updated_at":"2015-03-16T18:18:28Z","closed_at":null,"body":"Hi, \r\n\r\nI have some concerns about what is currently implemented as tf-idf in https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/text.py -- i.e., tf-idf = tf * (idf + 1).\r\n\r\nMy concern is regarding the \"+ 1\". The comment in the source code says, \"The actual formula used for tf-idf is tf * (idf + 1) = tf + tf * idf, instead of tf * idf. The effect of this is that terms with zero idf, i.e.  that occur in all documents of a training set, will not be entirely ignored.\" This is indeed true -- the computation is effectively linearly interpolating between tf * idf and tf -- however, adding one to idf is not standard practice. Even after extensive search, reading many IR and NLP papers regarding tf-idf, and consulting w/ IR and NLP researchers, I have not (yet!) encountered an instance of anyone doing this. I am therefore concerned that this is the default behavior in sklearn -- all the more so because there is no mention of this in the documentation (I had to read the source code to figure it out).\r\n\r\nAlthough there are several standard variants for computing tf and for computing idf (these are actually pretty well documented on the <a href=\"http://en.wikipedia.org/wiki/Tf%E2%80%93idf\">Wikipedia page</a>), they are always combined as tf * idf, not tf * (idf + 1). Granted (and taking the standard definition of idf for simplicity), you can subsume the \"+ 1\" into the idf score itself, giving idf + 1 = log(D / D_v) + 1 = log(D / D_v) + log(e) = log(e * D / D_v) -- in effect this means you're pretending you saw e times as many documents as you actually did, and none of them contained any terms in the vocabulary (this means that all terms -- frequent and infrequent -- will be treated as if they occurred in fewer documents than they actually did). However, this is not equivalent to any of the standard variants of idf.\r\n\r\nSince sklearn's tf-idf is widely used, I think it should reflect standard behavior/definitions in the IR and NLP literature, and thus the \"+ 1\" should be removed. To be clear, I'm happy for the \"+ 1\" to be left as an option (or even changed to be \"+ c\" with c speficied by the user), thereby allowing the user to linearly interpolate between tf and \"standard\" tf-idf, but I think users should be aware that they are doing this, rather than doing this and thinking they're using \"standard\" tf-idf."},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/81851156","html_url":"https://github.com/scikit-learn/scikit-learn/issues/4391#issuecomment-81851156","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4391","id":81851156,"user":{"login":"amueller","id":449558,"avatar_url":"https://avatars.githubusercontent.com/u/449558?v=3","gravatar_id":"","url":"https://api.github.com/users/amueller","html_url":"https://github.com/amueller","followers_url":"https://api.github.com/users/amueller/followers","following_url":"https://api.github.com/users/amueller/following{/other_user}","gists_url":"https://api.github.com/users/amueller/gists{/gist_id}","starred_url":"https://api.github.com/users/amueller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amueller/subscriptions","organizations_url":"https://api.github.com/users/amueller/orgs","repos_url":"https://api.github.com/users/amueller/repos","events_url":"https://api.github.com/users/amueller/events{/privacy}","received_events_url":"https://api.github.com/users/amueller/received_events","type":"User","site_admin":false},"created_at":"2015-03-16T18:18:28Z","updated_at":"2015-03-16T18:18:28Z","body":"oh, you are right. that is what the option does."}}