{"action":"created","issue":{"url":"https://api.github.com/repos/pystruct/pystruct/issues/129","labels_url":"https://api.github.com/repos/pystruct/pystruct/issues/129/labels{/name}","comments_url":"https://api.github.com/repos/pystruct/pystruct/issues/129/comments","events_url":"https://api.github.com/repos/pystruct/pystruct/issues/129/events","html_url":"https://github.com/pystruct/pystruct/pull/129","id":53201241,"number":129,"title":"use MemmapingPool instead of joblib.Parallel","user":{"login":"robbymeals","id":1600606,"avatar_url":"https://avatars.githubusercontent.com/u/1600606?v=3","gravatar_id":"","url":"https://api.github.com/users/robbymeals","html_url":"https://github.com/robbymeals","followers_url":"https://api.github.com/users/robbymeals/followers","following_url":"https://api.github.com/users/robbymeals/following{/other_user}","gists_url":"https://api.github.com/users/robbymeals/gists{/gist_id}","starred_url":"https://api.github.com/users/robbymeals/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robbymeals/subscriptions","organizations_url":"https://api.github.com/users/robbymeals/orgs","repos_url":"https://api.github.com/users/robbymeals/repos","events_url":"https://api.github.com/users/robbymeals/events{/privacy}","received_events_url":"https://api.github.com/users/robbymeals/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":51,"created_at":"2014-12-31T20:09:24Z","updated_at":"2015-01-07T16:36:26Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/pystruct/pystruct/pulls/129","html_url":"https://github.com/pystruct/pystruct/pull/129","diff_url":"https://github.com/pystruct/pystruct/pull/129.diff","patch_url":"https://github.com/pystruct/pystruct/pull/129.patch"},"body":"Hi! So using Parallel causes the respawning of pools with every iteration, which can be slow from the start, causes memory issues and is very slow once the matrices get to be at all large.  @amueller what do you think of this approach? I am sure there are other moving parts that I'm not aware of that need to be considered, but replacing the Parallel calls as in this PR in both OneSlackSSVM and SubgradientSSVM worked really well for me."},"comment":{"url":"https://api.github.com/repos/pystruct/pystruct/issues/comments/69048914","html_url":"https://github.com/pystruct/pystruct/pull/129#issuecomment-69048914","issue_url":"https://api.github.com/repos/pystruct/pystruct/issues/129","id":69048914,"user":{"login":"robbymeals","id":1600606,"avatar_url":"https://avatars.githubusercontent.com/u/1600606?v=3","gravatar_id":"","url":"https://api.github.com/users/robbymeals","html_url":"https://github.com/robbymeals","followers_url":"https://api.github.com/users/robbymeals/followers","following_url":"https://api.github.com/users/robbymeals/following{/other_user}","gists_url":"https://api.github.com/users/robbymeals/gists{/gist_id}","starred_url":"https://api.github.com/users/robbymeals/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robbymeals/subscriptions","organizations_url":"https://api.github.com/users/robbymeals/orgs","repos_url":"https://api.github.com/users/robbymeals/repos","events_url":"https://api.github.com/users/robbymeals/events{/privacy}","received_events_url":"https://api.github.com/users/robbymeals/received_events","type":"User","site_admin":false},"created_at":"2015-01-07T16:36:26Z","updated_at":"2015-01-07T16:36:26Z","body":"https://twitter.com/RobbyMeals/status/552649165835337728\r\n\r\nsolicited this reply from the guy who ported multiprocessing to python 2.7, and I think it sorta confirms my thinking, that the issue is the references:\r\n\r\n\"The reference counter in PyObject structs doesn't play well with copy-on-write fork(). Any write causes a copy of a memory page of 4096 bytes. I don't know how NumPy stores its arrays internally. A r/o view of shared memory may help\"\r\n\r\ndoesn't really explain the increased memory footprint before any writes have occurred though. Oh well. Maybe someone will answer the so question."}}