{"action":"created","issue":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/826","labels_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/826/labels{/name}","comments_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/826/comments","events_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/826/events","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/826","id":114105120,"number":826,"title":"Word2Vec exessive memory usage","user":{"login":"raver119","id":12250879,"avatar_url":"https://avatars.githubusercontent.com/u/12250879?v=3","gravatar_id":"","url":"https://api.github.com/users/raver119","html_url":"https://github.com/raver119","followers_url":"https://api.github.com/users/raver119/followers","following_url":"https://api.github.com/users/raver119/following{/other_user}","gists_url":"https://api.github.com/users/raver119/gists{/gist_id}","starred_url":"https://api.github.com/users/raver119/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raver119/subscriptions","organizations_url":"https://api.github.com/users/raver119/orgs","repos_url":"https://api.github.com/users/raver119/repos","events_url":"https://api.github.com/users/raver119/events{/privacy}","received_events_url":"https://api.github.com/users/raver119/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-10-29T17:37:13Z","updated_at":"2015-11-04T08:04:28Z","closed_at":null,"body":"I've noticed that modelling w2v, has exessive memory usage.\r\nModel description:\r\n3GB of text\r\n42 millions sentences.\r\n3.1 millions of words in Vocab.\r\n\r\nJVM settings:\r\n-Xms20G -Xmx20G\r\n\r\nVersions used: \r\nnd4j 0.4-rc3.5\r\ndl4j 0.4-rc3.4\r\n\r\n\r\nCode:\r\nTo reproduce this i've used Word2Vec example provided at dl4j-0.4-examples, with following params:\r\n.minWordFrequency(10)\r\n.iterations(3)\r\n.layerSize(200).\r\n\r\n\r\nExsessive memory usage exists in two calculation phases:\r\n1. initial Vocab buildiing\r\n2. w2v calulations\r\n\r\n\r\n# initial Vocab building\r\n\r\nHere's memory sampling screenshot taken at first phase:\r\nhttp://www.image123.net/v3eemxn70hsjpic.html\r\n\r\nAnd here's memory sampling screenshot taken right at the end of first phase:\r\nhttp://www.image123.net/nkqffiex4v21pic.html\r\n\r\nHere's overall graphs of JVisualVM:\r\nhttp://www.image123.net/mmk7rrznu9zypic.html\r\n\r\nIt looks like all raw content is stored untill the end of this phase, it feels like if you want to vectorize 100M lines, you'll have to store everything in RAM. And this will effectively prevent building models from large corpus.\r\n\r\n# w2v calculations\r\n\r\nHere's overall graphs that confirm memory release after Vocab is build in first phase:\r\nhttp://www.image123.net/uq2qgep64gkmpic.html\r\n\r\nAfter Vocab is build, memory usage drops down, so it looks like GC is not an issue here. As calculations go through words, memory usage starts growing, and grows up to OOM at 20G RAM used.\r\n\r\n\r\nGC overhead/OOM stack traces:\r\nhttps://gist.github.com/raver119/9048ef361aef5e409896\r\nhttps://gist.github.com/raver119/225cd1e4908cf25d2d44\r\n\r\nOverall graph right before OOM:\r\nhttp://www.image123.net/k4rw5ph7y9r1pic.html\r\n\r\nMemory sampling right before OOM:\r\nhttp://www.image123.net/yj5j7gtjyir8pic.html\r\n"},"comment":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/comments/153627214","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/826#issuecomment-153627214","issue_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/826","id":153627214,"user":{"login":"raver119","id":12250879,"avatar_url":"https://avatars.githubusercontent.com/u/12250879?v=3","gravatar_id":"","url":"https://api.github.com/users/raver119","html_url":"https://github.com/raver119","followers_url":"https://api.github.com/users/raver119/followers","following_url":"https://api.github.com/users/raver119/following{/other_user}","gists_url":"https://api.github.com/users/raver119/gists{/gist_id}","starred_url":"https://api.github.com/users/raver119/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raver119/subscriptions","organizations_url":"https://api.github.com/users/raver119/orgs","repos_url":"https://api.github.com/users/raver119/repos","events_url":"https://api.github.com/users/raver119/events{/privacy}","received_events_url":"https://api.github.com/users/raver119/received_events","type":"User","site_admin":false},"created_at":"2015-11-04T08:04:28Z","updated_at":"2015-11-04T08:04:28Z","body":"i doubt its a good idea, since at least the first part of this problem - vocab building - can be done \"on the fly\", without storiing everything in RAM. So, it doesn't needs any magic with vector-space.\r\n\r\nSecond part of this problem, that comes from overall memory usage at w2v calculations: this model in any circumstances will use (2 * vocab size * vector dimensions * data type size) memory. So no way any vector-space magic will reduce memory usage below this basic line. So, the task here is to find the way to stick to this basic memory usage, and stay as close to it as possible. Though, no idea how to make it so far :("}}