{"action":"created","issue":{"url":"https://api.github.com/repos/lmj/lparallel/issues/18","labels_url":"https://api.github.com/repos/lmj/lparallel/issues/18/labels{/name}","comments_url":"https://api.github.com/repos/lmj/lparallel/issues/18/comments","events_url":"https://api.github.com/repos/lmj/lparallel/issues/18/events","html_url":"https://github.com/lmj/lparallel/issues/18","id":72519295,"number":18,"title":"Waiting for completion of all pmap tasks before transferring an error","user":{"login":"i-perminov","id":1163115,"avatar_url":"https://avatars.githubusercontent.com/u/1163115?v=3","gravatar_id":"","url":"https://api.github.com/users/i-perminov","html_url":"https://github.com/i-perminov","followers_url":"https://api.github.com/users/i-perminov/followers","following_url":"https://api.github.com/users/i-perminov/following{/other_user}","gists_url":"https://api.github.com/users/i-perminov/gists{/gist_id}","starred_url":"https://api.github.com/users/i-perminov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/i-perminov/subscriptions","organizations_url":"https://api.github.com/users/i-perminov/orgs","repos_url":"https://api.github.com/users/i-perminov/repos","events_url":"https://api.github.com/users/i-perminov/events{/privacy}","received_events_url":"https://api.github.com/users/i-perminov/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-05-01T20:45:17Z","updated_at":"2015-05-03T21:54:56Z","closed_at":null,"body":"Hi,\r\n\r\nI have a case where I want to use pmap with a function that\r\nA. can signal an error\r\nB. is not interrupt-safe\r\nC. is not thread-safe when called with the same argument, i.e. it is ok to run (f x) and (f y) in parallel only if x /= y.\r\nI am not sure how to deal with errors in this case. If a task signals an error, the error gets transferred \r\nto the caller and the caller can run the same pmap again, which is not thread-safe (because of C).\r\nWrapping pmacar call with with-kill-on-abort is unsafe, because my function is not interrupt-safe.\r\nSo, it looks like I need to wait for completion of all the pmap tasks before transferring the error.\r\nIt would be nice to have macro with-wait-on-abort (similar to with-kill-on-abort), but I do not see an easy way to implement such a macro on top of pmap. Any suggestions?\r\n\r\nThanks,\r\nIlya"},"comment":{"url":"https://api.github.com/repos/lmj/lparallel/issues/comments/98549255","html_url":"https://github.com/lmj/lparallel/issues/18#issuecomment-98549255","issue_url":"https://api.github.com/repos/lmj/lparallel/issues/18","id":98549255,"user":{"login":"lmj","id":711363,"avatar_url":"https://avatars.githubusercontent.com/u/711363?v=3","gravatar_id":"","url":"https://api.github.com/users/lmj","html_url":"https://github.com/lmj","followers_url":"https://api.github.com/users/lmj/followers","following_url":"https://api.github.com/users/lmj/following{/other_user}","gists_url":"https://api.github.com/users/lmj/gists{/gist_id}","starred_url":"https://api.github.com/users/lmj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmj/subscriptions","organizations_url":"https://api.github.com/users/lmj/orgs","repos_url":"https://api.github.com/users/lmj/repos","events_url":"https://api.github.com/users/lmj/events{/privacy}","received_events_url":"https://api.github.com/users/lmj/received_events","type":"User","site_admin":false},"created_at":"2015-05-03T21:54:56Z","updated_at":"2015-05-03T21:54:56Z","body":"Since the restart doesn't really match the semantics of `cl:continue`, it would be better to name it `defer-error` (which also mnemonically matches `transfer-error`). Invoking `defer-error` would receive the remaining results while discarding any errors, then re-signal the original error. `with-deferred-errors` would signal at most once, at the end. Perhaps `defer-error` would just be an implementation detail.\r\n\r\nThough the implementation is straightforward (the only change to pmap et al would be using a channel subclass), `with-deferred-errors` may be too much. It occurred to me that you can also wait on tasks manually.\r\n\r\n    (defun test ()\r\n      (let ((q (lparallel.queue:make-queue))\r\n            (data (map-into (make-array 100) (let ((i -1))\r\n                                               (lambda () (incf i))))))\r\n        (handler-bind ((simple-error (lambda (e)\r\n                                       (declare (ignore e))\r\n                                       (loop repeat (1- (length data))\r\n                                             do (lparallel.queue:pop-queue q)))))\r\n          (lparallel:task-handler-bind ((error #'lparallel:invoke-transfer-error))\r\n            (lparallel:pmap 'vector\r\n                            (lambda (x)\r\n                              (when (zerop x)\r\n                                (error \"zero!\"))\r\n                              (lparallel.queue:push-queue t q)\r\n                              x)\r\n                            :parts (length data)\r\n                            data)))))\r\n\r\nA general solution for any `:parts` value would be more complicated, but if the calculation for each element in the sequence is heavy then setting the number of parts to the number of elements is already ideal."}}