{"action":"created","issue":{"url":"https://api.github.com/repos/piskvorky/gensim/issues/356","labels_url":"https://api.github.com/repos/piskvorky/gensim/issues/356/labels{/name}","comments_url":"https://api.github.com/repos/piskvorky/gensim/issues/356/comments","events_url":"https://api.github.com/repos/piskvorky/gensim/issues/356/events","html_url":"https://github.com/piskvorky/gensim/pull/356","id":87146225,"number":356,"title":"big doc-vector refactor/enhancements","user":{"login":"gojomo","id":1641,"avatar_url":"https://avatars.githubusercontent.com/u/1641?v=3","gravatar_id":"","url":"https://api.github.com/users/gojomo","html_url":"https://github.com/gojomo","followers_url":"https://api.github.com/users/gojomo/followers","following_url":"https://api.github.com/users/gojomo/following{/other_user}","gists_url":"https://api.github.com/users/gojomo/gists{/gist_id}","starred_url":"https://api.github.com/users/gojomo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gojomo/subscriptions","organizations_url":"https://api.github.com/users/gojomo/orgs","repos_url":"https://api.github.com/users/gojomo/repos","events_url":"https://api.github.com/users/gojomo/events{/privacy}","received_events_url":"https://api.github.com/users/gojomo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-06-10T23:43:36Z","updated_at":"2015-06-28T00:00:04Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/piskvorky/gensim/pulls/356","html_url":"https://github.com/piskvorky/gensim/pull/356","diff_url":"https://github.com/piskvorky/gensim/pull/356.diff","patch_url":"https://github.com/piskvorky/gensim/pull/356.patch"},"body":"Ready for review/testing!\r\n\r\nThe headline changes are optimized doc-vector inference, and separating doc-vectors (during training or comparison) from the word-vocabulary – allowing many more docs (via memmap-backing) than words, and avoiding some confusion.\r\n\r\nMany smaller changes include a DM-concatenative-context mode (as recommended in the Paragraph Vectors paper), an optional 'lock_factor' to attenuate training of some vectors, and other optimizations/cleanup.\r\n\r\nSee gensim/test/test_doc2vec.py to do a quick check on a new system or survey some API possibilities. See docs/notebook/doc2vec-IMDB.ipynb for a walkthrough of reproducing the PV paper IMDB sentiment experiment. \r\n\r\nIf you used Doc2Vec previously, a few key changes to note: \r\n* `LabeledSentence` is now `TaggedDocument`, and the (one or more) document labels that correspond to vectors are now referred to as 'tags'. Greatest memory efficiency is possible by using only int tags, contiguous and ascending from 0.\r\n* Doc vectors are stored, accessed, and compared through a consituent '.docvecs' field of the Doc2Vec model, rather than the old accessors/comparison methods that still work for words. So, `d2v_model.docvecs[doc_tag]` or `d2v_model.docvecs.most_similar(doc_tag)`, rather than on `d2v_model` directly.\r\n\r\nSome top needs:\r\n* testing on diverse, larger datasets and systems - though the ability to handle doc-vector sets much larger than RAM is theoretically there, I haven't forced an overflow yet\r\n* improving save() to externalize numpy arrays as with the prior models – maybe a recursive utils.SaveLoad?\r\n* refactoring the DocvecsArray similarity-testing methods – currently, just a quick-and-dirty copy-and-adapt from Word2Vec, but would ideally share code (and the optimizations pending in other @sebastien-j /  @KCzar PRs) with word-vecs"},"comment":{"url":"https://api.github.com/repos/piskvorky/gensim/issues/comments/116166847","html_url":"https://github.com/piskvorky/gensim/pull/356#issuecomment-116166847","issue_url":"https://api.github.com/repos/piskvorky/gensim/issues/356","id":116166847,"user":{"login":"gojomo","id":1641,"avatar_url":"https://avatars.githubusercontent.com/u/1641?v=3","gravatar_id":"","url":"https://api.github.com/users/gojomo","html_url":"https://github.com/gojomo","followers_url":"https://api.github.com/users/gojomo/followers","following_url":"https://api.github.com/users/gojomo/following{/other_user}","gists_url":"https://api.github.com/users/gojomo/gists{/gist_id}","starred_url":"https://api.github.com/users/gojomo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gojomo/subscriptions","organizations_url":"https://api.github.com/users/gojomo/orgs","repos_url":"https://api.github.com/users/gojomo/repos","events_url":"https://api.github.com/users/gojomo/events{/privacy}","received_events_url":"https://api.github.com/users/gojomo/received_events","type":"User","site_admin":false},"created_at":"2015-06-28T00:00:04Z","updated_at":"2015-06-28T00:00:04Z","body":"Thanks for the testing & review! Especially if your use-case is extreme in any interesting dimension – data volume, parameter choices, etc – it'd be great to hear confirmation things are working and useful (or alternatively, any ways that it's hitting limits). "}}