{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5122","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5122/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5122/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5122/events","html_url":"https://github.com/scikit-learn/scikit-learn/pull/5122","id":101165576,"number":5122,"title":"[WIP] Text vectorizers memory usage","user":{"login":"ephes","id":499500,"avatar_url":"https://avatars.githubusercontent.com/u/499500?v=3","gravatar_id":"","url":"https://api.github.com/users/ephes","html_url":"https://github.com/ephes","followers_url":"https://api.github.com/users/ephes/followers","following_url":"https://api.github.com/users/ephes/following{/other_user}","gists_url":"https://api.github.com/users/ephes/gists{/gist_id}","starred_url":"https://api.github.com/users/ephes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ephes/subscriptions","organizations_url":"https://api.github.com/users/ephes/orgs","repos_url":"https://api.github.com/users/ephes/repos","events_url":"https://api.github.com/users/ephes/events{/privacy}","received_events_url":"https://api.github.com/users/ephes/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-08-15T11:04:17Z","updated_at":"2015-11-04T00:38:45Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5122","html_url":"https://github.com/scikit-learn/scikit-learn/pull/5122","diff_url":"https://github.com/scikit-learn/scikit-learn/pull/5122.diff","patch_url":"https://github.com/scikit-learn/scikit-learn/pull/5122.patch"},"body":"Hi,\r\n\r\nthe other day I've tried to vectorize some big text data and wondered why\r\nsklean would use that much memory. So I looked into it and think I've found\r\nsome points.\r\n\r\nFor testing purposes I used the [enron data set](http://www.cs.cmu.edu/~enron/ \"enron data set\"). To reproduce my results without\r\nhaving to modify sklearn, you can use this little [script](https://gist.github.com/ephes/90193567b8c0501b13ef \"enron test script\") (now including the vectorizer\r\n from #4968 ). Just unpack the enron dataset and start the script with:\r\n\r\n```\r\ntime python test_enron.py ~/data/enron/maildir enron_features.dump\r\n```\r\n\r\nHere's a little table showing the results. CountVectorizer and HashingVectorizer\r\nare the unmodified vectorizers from sklearn. LeanCountVectorizer and\r\nLeanHashingVectorizer are the new modified versions. AupiffCountVectorizer\r\nis the modified version from #4968 inheriting from LeanCountVectorizer (from\r\nCountVectorizer in parantheses). FastAupiffCountVectorizer uses 'if else' instead\r\nof .get in inner loop and .keys() and .values insted of iteritems + an explicit\r\ndel(feature_counter) after adding a new doc:\r\n\r\n| Vectorizer            | after 300K | after 510K | peak    | time    |\r\n| --------------------- |:----------:| ----------:|---------|---------|\r\n| CountVectorizer       | 0.43 GB    | 0.84 GB    | 5.14 GB | 237,24s |\r\n| LeanCountVectorizer   | 1.08 GB    | 1.41 GB    | 1.87 GB | 268,19s | \r\n| AupiffCountVectorizer | 0.41 GB | 0.8 GB | 1.86 GB (3.3 GB) | 287,43s (280,02s) |\r\n| FastAupiffCountVectorizer | 0.27 GB | 0.43 GB | 1.69 GB | 253,92s |\r\n| HashingVectorizer     | 1.46 GB    | 2.75 GB    | 2.82 GB | 243,02s |\r\n| LeanHashingVectorizer | 1.28 GB    | 1.55 GB    | 1.64 GB | 232,62s |\r\n\r\nWhy is the default CountVectorizer using 5 GB memory to produce a\r\nfeature matrix which is only 1.15 GB in size when dumped to disc via joblib?\r\nMaybe it's the vocabulary? Let's see:\r\n\r\n```\r\nThe feature matrix consists of the numpy arrays indices, indptr and values:\r\nindices size:  835.65 MB\r\nindptr size :    2.07 MB\r\nvalues size : 1671.23 MB\r\n\r\nSize of the feature matrix before X.sum_duplicates() is called:\r\n2509 MB\r\n\r\nSize of the feature matrix after X.sum_duplicates() was called:\r\n1147 MB\r\n\r\nvocabulary size: 100.66 MB\r\n```\r\n\r\nWell, it's not the vocabulary, it's the feature matrix. And if\r\nusing 2.5 GB is not bad enough, either X.sum_duplicates or\r\nsp.csr_matrix are copying some data, because I saw 4.25 GB memory\r\nusage even with an sys.exit(0) at the end of _count_vocab.\r\n\r\nIn the LeanCountVectorizer I iterate chunk-wise over the documents,\r\ncreate a temporary feature matrix for each chunk and append this\r\ntemporary feature matrix then to the final feature matrix in place.\r\nSo there's never this big values array and the data gets also copied,\r\nbut only for the small chunk based feature matrices.\r\n\r\nAfter _count_vocab returned the feature matrix, the method _sort_features\r\nis called. Although it modifies the vocabulary in place as stated in\r\nthe docstring, it does not modify the feature matrix in place, but\r\nmakes a copy via fancy indexing (return X[:, map_index]). So I changed\r\nthe _sort_features method for LeanCountVectorizer to modify the feature\r\nmatrix in place. This operation is rather slow and the main reason\r\nwhy LeanCountVectorizer is slower than CountVectorizer. Maybe there's\r\na better way to swap the columns of the feature matrix.\r\n\r\nAnd finally the _limit_features method also makes a full copy of the\r\nfeature matrix via fancy indexing, because it's high and low arguments\r\nare set to numerical values per default so the 'if high is None and...'\r\ncondition is always false. In LeanCountVectorizer this is also changed.\r\n\r\nThe LeanHashingVectorizer has only the 'iterate in chunks over documents'\r\nmodification. And since the values array would not get as big, it uses\r\nless memory.\r\n\r\nbest regards,\r\nJochen"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/153534506","html_url":"https://github.com/scikit-learn/scikit-learn/pull/5122#issuecomment-153534506","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5122","id":153534506,"user":{"login":"aupiff","id":804266,"avatar_url":"https://avatars.githubusercontent.com/u/804266?v=3","gravatar_id":"","url":"https://api.github.com/users/aupiff","html_url":"https://github.com/aupiff","followers_url":"https://api.github.com/users/aupiff/followers","following_url":"https://api.github.com/users/aupiff/following{/other_user}","gists_url":"https://api.github.com/users/aupiff/gists{/gist_id}","starred_url":"https://api.github.com/users/aupiff/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aupiff/subscriptions","organizations_url":"https://api.github.com/users/aupiff/orgs","repos_url":"https://api.github.com/users/aupiff/repos","events_url":"https://api.github.com/users/aupiff/events{/privacy}","received_events_url":"https://api.github.com/users/aupiff/received_events","type":"User","site_admin":false},"created_at":"2015-11-04T00:38:44Z","updated_at":"2015-11-04T00:38:44Z","body":"@jnothman yes."}}