{"action":"created","issue":{"url":"https://api.github.com/repos/BVLC/caffe/issues/1377","labels_url":"https://api.github.com/repos/BVLC/caffe/issues/1377/labels{/name}","comments_url":"https://api.github.com/repos/BVLC/caffe/issues/1377/comments","events_url":"https://api.github.com/repos/BVLC/caffe/issues/1377/events","html_url":"https://github.com/BVLC/caffe/issues/1377","id":47186189,"number":1377,"title":"Caffe memory increases with time(iterations?)","user":{"login":"lavania","id":6517878,"avatar_url":"https://avatars.githubusercontent.com/u/6517878?v=3","gravatar_id":"","url":"https://api.github.com/users/lavania","html_url":"https://github.com/lavania","followers_url":"https://api.github.com/users/lavania/followers","following_url":"https://api.github.com/users/lavania/following{/other_user}","gists_url":"https://api.github.com/users/lavania/gists{/gist_id}","starred_url":"https://api.github.com/users/lavania/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lavania/subscriptions","organizations_url":"https://api.github.com/users/lavania/orgs","repos_url":"https://api.github.com/users/lavania/repos","events_url":"https://api.github.com/users/lavania/events{/privacy}","received_events_url":"https://api.github.com/users/lavania/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2014-10-29T18:30:51Z","updated_at":"2015-07-16T05:53:18Z","closed_at":null,"body":"So I have been trying to train the imagenet example using caffe. I am using cuda 6.5 with cuDNN. But when I run the trainer as shown in http://caffe.berkeleyvision.org/gathered/examples/imagenet.html along with GLOG_log_dir I see that the memory usage of caffe (seen using top) keeps on increasing with time (iterations ?) . So the top gives something like :\r\n\r\nSIZE   RES   SHR STATE   TIME    CPU COMMAND\r\n2285G   58G   57G run    28:57   140% caffe\r\n\r\nThis causes the machine to slow down to a crawl (cannot type anything on the console). Any idea what might be causing this. I see this with both leveldb and lmdb.\r\n\r\nNote that the GPU memory usage remains fixed.\r\n\r\nRegards\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/BVLC/caffe/issues/comments/121841242","html_url":"https://github.com/BVLC/caffe/issues/1377#issuecomment-121841242","issue_url":"https://api.github.com/repos/BVLC/caffe/issues/1377","id":121841242,"user":{"login":"woozzu","id":1217229,"avatar_url":"https://avatars.githubusercontent.com/u/1217229?v=3","gravatar_id":"","url":"https://api.github.com/users/woozzu","html_url":"https://github.com/woozzu","followers_url":"https://api.github.com/users/woozzu/followers","following_url":"https://api.github.com/users/woozzu/following{/other_user}","gists_url":"https://api.github.com/users/woozzu/gists{/gist_id}","starred_url":"https://api.github.com/users/woozzu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/woozzu/subscriptions","organizations_url":"https://api.github.com/users/woozzu/orgs","repos_url":"https://api.github.com/users/woozzu/repos","events_url":"https://api.github.com/users/woozzu/events{/privacy}","received_events_url":"https://api.github.com/users/woozzu/received_events","type":"User","site_admin":false},"created_at":"2015-07-16T05:53:18Z","updated_at":"2015-07-16T05:53:18Z","body":"For Windows case, the problem of lmdb can be solved by just adding following lines at the top of LMDBCursor::Seek method. It releases memory mapped pages which are not used after current seek.\r\n\r\n```\r\nif (op != MDB_FIRST)\r\n    VirtualUnlock(mdb_value_.mv_data, mdb_value_.mv_size);\r\n```"}}