{"action":"created","issue":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/3836","labels_url":"https://api.github.com/repos/JuliaLang/julia/issues/3836/labels{/name}","comments_url":"https://api.github.com/repos/JuliaLang/julia/issues/3836/comments","events_url":"https://api.github.com/repos/JuliaLang/julia/issues/3836/events","html_url":"https://github.com/JuliaLang/julia/issues/3836","id":17254958,"number":3836,"title":"Heartbeats between master and worker processes. ","user":{"login":"amitmurthy","id":117461,"avatar_url":"https://avatars.githubusercontent.com/u/117461?v=3","gravatar_id":"","url":"https://api.github.com/users/amitmurthy","html_url":"https://github.com/amitmurthy","followers_url":"https://api.github.com/users/amitmurthy/followers","following_url":"https://api.github.com/users/amitmurthy/following{/other_user}","gists_url":"https://api.github.com/users/amitmurthy/gists{/gist_id}","starred_url":"https://api.github.com/users/amitmurthy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amitmurthy/subscriptions","organizations_url":"https://api.github.com/users/amitmurthy/orgs","repos_url":"https://api.github.com/users/amitmurthy/repos","events_url":"https://api.github.com/users/amitmurthy/events{/privacy}","received_events_url":"https://api.github.com/users/amitmurthy/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/JuliaLang/julia/labels/parallel","name":"parallel","color":"f7e102"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/JuliaLang/julia/milestones/8","html_url":"https://github.com/JuliaLang/julia/milestones/0.4.x","labels_url":"https://api.github.com/repos/JuliaLang/julia/milestones/8/labels","id":303801,"number":8,"title":"0.4.x","description":"Bugs to fix for some 0.4.x release.","creator":{"login":"vtjnash","id":330950,"avatar_url":"https://avatars.githubusercontent.com/u/330950?v=3","gravatar_id":"","url":"https://api.github.com/users/vtjnash","html_url":"https://github.com/vtjnash","followers_url":"https://api.github.com/users/vtjnash/followers","following_url":"https://api.github.com/users/vtjnash/following{/other_user}","gists_url":"https://api.github.com/users/vtjnash/gists{/gist_id}","starred_url":"https://api.github.com/users/vtjnash/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vtjnash/subscriptions","organizations_url":"https://api.github.com/users/vtjnash/orgs","repos_url":"https://api.github.com/users/vtjnash/repos","events_url":"https://api.github.com/users/vtjnash/events{/privacy}","received_events_url":"https://api.github.com/users/vtjnash/received_events","type":"User","site_admin":false},"open_issues":50,"closed_issues":110,"state":"open","created_at":"2013-04-05T16:15:28Z","updated_at":"2015-10-17T04:29:56Z","due_on":null,"closed_at":null},"comments":14,"created_at":"2013-07-26T09:20:42Z","updated_at":"2015-10-28T00:29:46Z","closed_at":null,"body":"Heartbeats between master and workers have been mentioned before. Seeing https://github.com/JuliaLang/julia-ipython/issues/8 triggered it again.\r\n\r\nSuggesting a design : \r\n\r\n- The requirement is to provide an alternate communication channel to a Julia worker process that can be used for 2 things presently - a) heartbeats and b) querying the state of the process (including long running computation progress)\r\n\r\n- This has to be in a different thread since a long running compute bound task will not yield frequently enough \r\n\r\n- The handler also needs to be in C since core Julia is not yet thread-safe\r\n\r\n- Currently the ```accept_handler``` in ```multi.jl``` waits for connection requests from peer processes.\r\n\r\n- Upon a new connection it will now read a a short 4-byte header field that will specify what type of connection it is. \r\n\r\n- If the client has sent \"WRKR\", it will call ```create_message_handler_loop``` as it currently does\r\n\r\n- If the client has sent \"ADMN\", it will disengage the socket from libuv, start a new OS thread and execute a C \"admin\" function in the new thread with this socket. Only one admin thread can be running at any time.\r\n\r\n- The \"admin\" function will currently just respond to \"ping\" messages with some state information about the worker process. Maybe also provide a means for  https://github.com/timholy/ProgressMeter.jl to publish compute progress information which is returned with every ping. Will use a pipe to push information between the Julia thread and the admin thread.\r\n\r\n- If a ping message has not been received for a specified timeout (say 2 minutes) or this socket connection breaks, the worker exits if configured to do so.\r\n\r\n- Julia master process too starts a new thread (C code)  that continually sends these heartbeat messages to all connected workers and exposes the returned state information to the Julia thread. A ```show_workers()``` can pretty print this information.\r\n\r\nThe whole heartbeat mechanism can be switched on via a command line argument to the julia executable (default is off)\r\n\r\nIt seems workable enough and we don't need the admin thread to listen on a different port. Do let me know what you guys think.\r\n"},"comment":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/comments/151683149","html_url":"https://github.com/JuliaLang/julia/issues/3836#issuecomment-151683149","issue_url":"https://api.github.com/repos/JuliaLang/julia/issues/3836","id":151683149,"user":{"login":"ssfrr","id":734386,"avatar_url":"https://avatars.githubusercontent.com/u/734386?v=3","gravatar_id":"","url":"https://api.github.com/users/ssfrr","html_url":"https://github.com/ssfrr","followers_url":"https://api.github.com/users/ssfrr/followers","following_url":"https://api.github.com/users/ssfrr/following{/other_user}","gists_url":"https://api.github.com/users/ssfrr/gists{/gist_id}","starred_url":"https://api.github.com/users/ssfrr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ssfrr/subscriptions","organizations_url":"https://api.github.com/users/ssfrr/orgs","repos_url":"https://api.github.com/users/ssfrr/repos","events_url":"https://api.github.com/users/ssfrr/events{/privacy}","received_events_url":"https://api.github.com/users/ssfrr/received_events","type":"User","site_admin":false},"created_at":"2015-10-28T00:29:46Z","updated_at":"2015-10-28T00:29:46Z","body":"I'll preface this by saying that I'm not entirely convinced either that this is a better way to go, and it's possible that my ignorance of the larger context is getting in the way. Here are a couple of things that concern me with the random delimiter scheme though:\r\n\r\n1. though the chance of a collision is almost infinitesimally small, it's nonzero and I imagine lots and lots of these messages getting sent. I'm not sure what the target is here for failure probability, but this is probably just me being pedantic.\r\n2. if the process writing the frame gets borked in a way that prevents it from writing the closing delimiter, the stream never recovers. This might not be a failure mode that we need to worry about though.\r\n\r\nRe: needing to know the number of frame bytes: that's why you'd need the 254-byte buffer on the writer side. You only need to lookahead for a `0x00` up to 254 bytes because if it's longer than that you just put `0xff` as the frame byte and send the next 254 bytes as-is.\r\n\r\nAnyways, mostly I just wanted to throw it out there as an alternative, not advocate to change course, so if these points aren't convincing I think the random delimiter is also a good way to go."}}