{"action":"opened","issue":{"url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/353","labels_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/353/labels{/name}","comments_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/353/comments","events_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/353/events","html_url":"https://github.com/spark-jobserver/spark-jobserver/issues/353","id":124290814,"number":353,"title":"Proper reconnection / HA for forked JVM contexts","user":{"login":"velvia","id":1062875,"avatar_url":"https://avatars.githubusercontent.com/u/1062875?v=3","gravatar_id":"","url":"https://api.github.com/users/velvia","html_url":"https://github.com/velvia","followers_url":"https://api.github.com/users/velvia/followers","following_url":"https://api.github.com/users/velvia/following{/other_user}","gists_url":"https://api.github.com/users/velvia/gists{/gist_id}","starred_url":"https://api.github.com/users/velvia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/velvia/subscriptions","organizations_url":"https://api.github.com/users/velvia/orgs","repos_url":"https://api.github.com/users/velvia/repos","events_url":"https://api.github.com/users/velvia/events{/privacy}","received_events_url":"https://api.github.com/users/velvia/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/labels/architecture","name":"architecture","color":"fbca04"},{"url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/milestones/4","html_url":"https://github.com/spark-jobserver/spark-jobserver/milestones/0.7.0","labels_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/milestones/4/labels","id":1477676,"number":4,"title":"0.7.0","description":"","creator":{"login":"velvia","id":1062875,"avatar_url":"https://avatars.githubusercontent.com/u/1062875?v=3","gravatar_id":"","url":"https://api.github.com/users/velvia","html_url":"https://github.com/velvia","followers_url":"https://api.github.com/users/velvia/followers","following_url":"https://api.github.com/users/velvia/following{/other_user}","gists_url":"https://api.github.com/users/velvia/gists{/gist_id}","starred_url":"https://api.github.com/users/velvia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/velvia/subscriptions","organizations_url":"https://api.github.com/users/velvia/orgs","repos_url":"https://api.github.com/users/velvia/repos","events_url":"https://api.github.com/users/velvia/events{/privacy}","received_events_url":"https://api.github.com/users/velvia/received_events","type":"User","site_admin":false},"open_issues":4,"closed_issues":0,"state":"open","created_at":"2015-12-30T00:19:00Z","updated_at":"2015-12-30T01:21:55Z","due_on":"2016-02-28T08:00:00Z","closed_at":null},"comments":0,"created_at":"2015-12-30T01:21:55Z","updated_at":"2015-12-30T01:21:55Z","closed_at":null,"body":"Right now with the JVM-per-context PR merged, here is one problem that remains.  When the forked JVM contexts are not explicitly killed, and a user runs `server_stop.sh`, then only the HTTP WebApi process is killed, not the child JVMs.  This is a problem because when the process is re-launched, it is not able to re-connect with the forked JVMs.  While the master HTTP server is down, the forked manager JVM is not able to persist status changes, because the JobDAO is only run on the HTTP server.\r\n\r\nWhile we could have the forked JVMs kill themselves when the master goes down, it seems much better to be able to re-start the HTTP component and re-connect to running contexts.   Thus the goals of this issue are:\r\n\r\n1. If the HTTP process (\"master\") goes down, and comes back up, it is able to discover currently running forked JVM contexts and automatically add them to the list of current contexts.  This may mean that contexts that are auto-started do not need to be started up anymore.\r\n1. If the forked JVM process goes down, or heartbeat dies, it is properly recognized by the HTTP master and marked as down\r\n1. If the forked JVM process goes down, and comes back up, the HTTP master recognizes it\r\n1. When the HTTP process (\"master\") is down, the forked JVM contexts should still be able to update status in the DB.\r\n1. All of the above should be testable in the regular test suite, probably using sbt-multi-jvm and the Akka clustering testing plugin.\r\n\r\nSome implications of the above:\r\n\r\n- Forked JVM processes must have their own JobDAO.  I would in fact eliminate the JobDAOActor.  This satisfies the ability to update when the HTTP master is down.\r\n- Discovery of forked JVMs implies that state about existing running contexts (as well as their port #'s or Akka connection info) must be persisted somewhere.\r\n- Need to investigate if we need to connect with just one member of the remaining cluster or how many (or all of them)."}}