{"action":"created","issue":{"url":"https://api.github.com/repos/PacificBiosciences/GenomicConsensus/issues/17","labels_url":"https://api.github.com/repos/PacificBiosciences/GenomicConsensus/issues/17/labels{/name}","comments_url":"https://api.github.com/repos/PacificBiosciences/GenomicConsensus/issues/17/comments","events_url":"https://api.github.com/repos/PacificBiosciences/GenomicConsensus/issues/17/events","html_url":"https://github.com/PacificBiosciences/GenomicConsensus/issues/17","id":88494725,"number":17,"title":"Running GenomicConsensus.Quiver.utils.consensusForAlignments() with multiprocessing generates all kinds of errors","user":{"login":"verheytb","id":10568362,"avatar_url":"https://avatars.githubusercontent.com/u/10568362?v=3","gravatar_id":"","url":"https://api.github.com/users/verheytb","html_url":"https://github.com/verheytb","followers_url":"https://api.github.com/users/verheytb/followers","following_url":"https://api.github.com/users/verheytb/following{/other_user}","gists_url":"https://api.github.com/users/verheytb/gists{/gist_id}","starred_url":"https://api.github.com/users/verheytb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/verheytb/subscriptions","organizations_url":"https://api.github.com/users/verheytb/orgs","repos_url":"https://api.github.com/users/verheytb/repos","events_url":"https://api.github.com/users/verheytb/events{/privacy}","received_events_url":"https://api.github.com/users/verheytb/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-06-15T17:50:27Z","updated_at":"2015-06-16T20:16:47Z","closed_at":null,"body":"I have written some Python code that basically calls on the consensusForAlignments() function from GenomicConsensus.Quiver.utils. I need to use this function a lot, so I'm trying to use multiprocessing.Pool to speed things up.\r\n\r\nWhen I call this function only once at a time, it returns as expected; However, when I call it from multiple worker processes using multiprocessing.Pool, I get a variety of errors including \"IOError: can't read data (Dataset: Read failed)\", \"RuntimeError: Invalid DelTag field---must be ASCII encoded.\", and \"KeyError: unable to open object (Symbol table: Can't open object)\".\r\n\r\nI thought that Python's multiprocessing has each worker function in separate memory space, so there shouldn't be any memory conflicts or the like that would cause something like that.\r\n\r\nAny ideas?"},"comment":{"url":"https://api.github.com/repos/PacificBiosciences/GenomicConsensus/issues/comments/112555520","html_url":"https://github.com/PacificBiosciences/GenomicConsensus/issues/17#issuecomment-112555520","issue_url":"https://api.github.com/repos/PacificBiosciences/GenomicConsensus/issues/17","id":112555520,"user":{"login":"dalexander","id":378620,"avatar_url":"https://avatars.githubusercontent.com/u/378620?v=3","gravatar_id":"","url":"https://api.github.com/users/dalexander","html_url":"https://github.com/dalexander","followers_url":"https://api.github.com/users/dalexander/followers","following_url":"https://api.github.com/users/dalexander/following{/other_user}","gists_url":"https://api.github.com/users/dalexander/gists{/gist_id}","starred_url":"https://api.github.com/users/dalexander/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dalexander/subscriptions","organizations_url":"https://api.github.com/users/dalexander/orgs","repos_url":"https://api.github.com/users/dalexander/repos","events_url":"https://api.github.com/users/dalexander/events{/privacy}","received_events_url":"https://api.github.com/users/dalexander/received_events","type":"User","site_admin":false},"created_at":"2015-06-16T20:16:47Z","updated_at":"2015-06-16T20:16:47Z","body":"That's the way it *should* work, but we historically have run into problems when using hdf5 in mutli-process operation.  IIRC, you need to make sure you don't have the hdf5 file open in the parent process before you fork any child processes."}}