{"action":"opened","issue":{"url":"https://api.github.com/repos/NLeSC/PattyGenerate/issues/28","labels_url":"https://api.github.com/repos/NLeSC/PattyGenerate/issues/28/labels{/name}","comments_url":"https://api.github.com/repos/NLeSC/PattyGenerate/issues/28/comments","events_url":"https://api.github.com/repos/NLeSC/PattyGenerate/issues/28/events","html_url":"https://github.com/NLeSC/PattyGenerate/issues/28","id":62630071,"number":28,"title":"Create/Find a multithreaded KeyMatchFull","user":{"login":"jmaassen","id":1705533,"avatar_url":"https://avatars.githubusercontent.com/u/1705533?v=3","gravatar_id":"","url":"https://api.github.com/users/jmaassen","html_url":"https://github.com/jmaassen","followers_url":"https://api.github.com/users/jmaassen/followers","following_url":"https://api.github.com/users/jmaassen/following{/other_user}","gists_url":"https://api.github.com/users/jmaassen/gists{/gist_id}","starred_url":"https://api.github.com/users/jmaassen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmaassen/subscriptions","organizations_url":"https://api.github.com/users/jmaassen/orgs","repos_url":"https://api.github.com/users/jmaassen/repos","events_url":"https://api.github.com/users/jmaassen/events{/privacy}","received_events_url":"https://api.github.com/users/jmaassen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-03-18T09:03:25Z","updated_at":"2015-03-18T09:03:25Z","closed_at":null,"body":"The current KeyMatchFull does a single threaded all-to-all compare of all image keysets. To parallalise this I created a KeyMatchPart which does a one-to-all compare. This KeyMatchPart can then easily be run in parallel using a python thread pool. \r\n\r\nUnfortunately, this approach increases the memory usage by a factor N (where N is the number of processes) since each process loads its own copy of the keys. Also, the key loading time is increased by a factor X/2 (where X is the number of images) since each process loads (part of) the keys over and over again. \r\n\r\nFor large datasets this will become a problem. "}}