{"action":"created","issue":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/12640","labels_url":"https://api.github.com/repos/JuliaLang/julia/issues/12640/labels{/name}","comments_url":"https://api.github.com/repos/JuliaLang/julia/issues/12640/comments","events_url":"https://api.github.com/repos/JuliaLang/julia/issues/12640/events","html_url":"https://github.com/JuliaLang/julia/issues/12640","id":101219790,"number":12640,"title":"Ctrl-C when master process is waiting for crashed workers","user":{"login":"alyst","id":348591,"avatar_url":"https://avatars.githubusercontent.com/u/348591?v=3","gravatar_id":"","url":"https://api.github.com/users/alyst","html_url":"https://github.com/alyst","followers_url":"https://api.github.com/users/alyst/followers","following_url":"https://api.github.com/users/alyst/following{/other_user}","gists_url":"https://api.github.com/users/alyst/gists{/gist_id}","starred_url":"https://api.github.com/users/alyst/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alyst/subscriptions","organizations_url":"https://api.github.com/users/alyst/orgs","repos_url":"https://api.github.com/users/alyst/repos","events_url":"https://api.github.com/users/alyst/events{/privacy}","received_events_url":"https://api.github.com/users/alyst/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/JuliaLang/julia/labels/parallel","name":"parallel","color":"f7e102"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-08-15T23:39:15Z","updated_at":"2015-09-09T00:35:31Z","closed_at":null,"body":"When master Julia process is waiting in `take!()` on its own `RemoteRef` and the worker processes have all thrown exceptions, pressing Ctrl-C in REPL results in\r\n```\r\nERROR (unhandled task failure): InterruptException:\r\n in task_done_hook at ./task.jl:175\r\n```\r\nbut doesn't bring the master from the waiting state."},"comment":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/comments/138739606","html_url":"https://github.com/JuliaLang/julia/issues/12640#issuecomment-138739606","issue_url":"https://api.github.com/repos/JuliaLang/julia/issues/12640","id":138739606,"user":{"login":"alyst","id":348591,"avatar_url":"https://avatars.githubusercontent.com/u/348591?v=3","gravatar_id":"","url":"https://api.github.com/users/alyst","html_url":"https://github.com/alyst","followers_url":"https://api.github.com/users/alyst/followers","following_url":"https://api.github.com/users/alyst/following{/other_user}","gists_url":"https://api.github.com/users/alyst/gists{/gist_id}","starred_url":"https://api.github.com/users/alyst/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alyst/subscriptions","organizations_url":"https://api.github.com/users/alyst/orgs","repos_url":"https://api.github.com/users/alyst/repos","events_url":"https://api.github.com/users/alyst/events{/privacy}","received_events_url":"https://api.github.com/users/alyst/received_events","type":"User","site_admin":false},"created_at":"2015-09-09T00:35:31Z","updated_at":"2015-09-09T00:35:31Z","body":"It was not as simple as I've described. In very simple examples Julia behaves normally.\r\nHowever, here's the stripped down version of master/slave parallel optimizer from robertfeldt/BlackBoxOptim.jl#25. Unfortunately it's not the most minimal test, but at least it should demonstrate the problem.\r\nThe master stucks in `step!()` waiting for the slave that have thrown an exception.\r\nWhen I run this code from REPL, Ctrl+C while waiting for `take!()` gives the above exception. From command line it's ok.\r\n```julia\r\naddprocs(1)\r\n\r\n@everywhere module TestTake\r\n\r\n# message with the candidate passed between the workers and the master\r\n\r\ntypealias WorkerChannel Channel{Float64}\r\ntypealias WorkerChannelRef RemoteRef{WorkerChannel}\r\n\r\ntype ParallelPopulationOptimizer\r\n  worker_procs::Vector{Int}                         # IDs of worker processes\r\n  final_fitnesses::Vector{RemoteRef{Channel{Any}}}  # references to the @spawnat ID run_worker()\r\n  from_workers::WorkerChannelRef                 # inbound channel of candidates from all workers\r\n  to_workers::Vector{WorkerChannelRef}           # outgoing channels to each worker\r\n  is_started::RemoteRef{Channel{Bool}}              # flag that all workers have started\r\nend\r\n\r\nnworkers(ppopt::ParallelPopulationOptimizer) = length(ppopt.worker_procs)\r\n\r\n# outer parallel population optimizer constructor that\r\n# also spawns worker tasks\r\nfunction ParallelPopulationOptimizer(NWorkers::Int = 2,\r\n    ArchiveCapacity::Int = 10,\r\n    ToWorkerChannelCapacity::Int = 1000,\r\n    FromWorkersChannelCapacity::Int = 10000)\r\n  # take the first NWorkers workers\r\n  Workers = workers()\r\n  ParallelPopulationOptimizer(Workers,\r\n       Vector{RemoteRef{Channel{Any}}}(length(Workers)),\r\n       RemoteRef(() -> WorkerChannel(FromWorkersChannelCapacity)),\r\n       WorkerChannelRef[RemoteRef(() -> WorkerChannel(ToWorkerChannelCapacity), id) for id in Workers],\r\n       RemoteRef(() -> Channel{Bool}(1)))\r\nend\r\n\r\nfunction setup!(ppopt::ParallelPopulationOptimizer)\r\n  info(\"Initializing parallel workers...\")\r\n  workers_ready = RemoteRef(() -> Channel{Int}(nworkers(ppopt))) # FIXME do we need to wait for the worker?\r\n  @assert !isready(ppopt.is_started)\r\n  for i in eachindex(ppopt.worker_procs)\r\n    procid = ppopt.worker_procs[i]\r\n    info(\"  Spawning worker #$i at process #$procid...\");\r\n    ppopt.final_fitnesses[i] = @spawnat procid run_worker(i,\r\n                           workers_ready, ppopt.is_started\r\n                           )\r\n  end\r\n  info(\"Waiting for the workers to be ready...\")\r\n  nready = 0\r\n  while nready < nworkers(ppopt)\r\n    worker_id = take!(workers_ready)\r\n    info(\"  Worker #$worker_id is ready\")\r\n    nready += 1\r\n  end\r\n  info(\"All workers ready\")\r\n  return ppopt\r\nend\r\n\r\nfunction step!(ppopt::ParallelPopulationOptimizer)\r\n  #println(\"main#: n_evals=$(num_evals(ppopt.evaluator))\")\r\n  if !isready(ppopt.is_started) put!(ppopt.is_started, true) end # if it's the first iteration\r\n  info(\"take!...\")\r\n  candidate = take!(ppopt.from_workers)#::CandidateMessage\r\n  info(\"take!\")\r\nend\r\n\r\nfunction run_worker(id::Int,\r\n                    worker_ready::RemoteRef{Channel{Int}},\r\n                    is_started::RemoteRef{Channel{Bool}} )\r\n  put!(worker_ready, id)\r\n  fetch(is_started) # wait until the master is started\r\n  try\r\n    error(\"just error\")\r\n  catch e\r\n    rethrow(e)\r\n  end\r\nend\r\n\r\nend\r\n\r\nppopt = TestTake.ParallelPopulationOptimizer()\r\nTestTake.setup!(ppopt)\r\nTestTake.step!(ppopt)\r\n```\r\n"}}