{"action":"created","issue":{"url":"https://api.github.com/repos/linkedin/gobblin/issues/155","labels_url":"https://api.github.com/repos/linkedin/gobblin/issues/155/labels{/name}","comments_url":"https://api.github.com/repos/linkedin/gobblin/issues/155/comments","events_url":"https://api.github.com/repos/linkedin/gobblin/issues/155/events","html_url":"https://github.com/linkedin/gobblin/pull/155","id":85330384,"number":155,"title":"Do not persist state of a job if its output data has not been committed","user":{"login":"liyinan926","id":3709137,"avatar_url":"https://avatars.githubusercontent.com/u/3709137?v=3","gravatar_id":"","url":"https://api.github.com/users/liyinan926","html_url":"https://github.com/liyinan926","followers_url":"https://api.github.com/users/liyinan926/followers","following_url":"https://api.github.com/users/liyinan926/following{/other_user}","gists_url":"https://api.github.com/users/liyinan926/gists{/gist_id}","starred_url":"https://api.github.com/users/liyinan926/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liyinan926/subscriptions","organizations_url":"https://api.github.com/users/liyinan926/orgs","repos_url":"https://api.github.com/users/liyinan926/repos","events_url":"https://api.github.com/users/liyinan926/events{/privacy}","received_events_url":"https://api.github.com/users/liyinan926/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-06-04T23:41:03Z","updated_at":"2015-06-05T04:21:26Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/linkedin/gobblin/pulls/155","html_url":"https://github.com/linkedin/gobblin/pull/155","diff_url":"https://github.com/linkedin/gobblin/pull/155.diff","patch_url":"https://github.com/linkedin/gobblin/pull/155.patch"},"body":"This pull request is motivated by a case from the Kafka adapter that doesn't want to have job state persisted if the job failed (so to be consistent with the Camus behavior). This pull request has the following changes (UPDATED):\r\n1. `AbstractJobLauncher` will no longer persist job state of a job if its output data has not been committed successfully.\r\n2. Added a check in `MRJobLauncher` on the number of `TaskState`s collected from the mapper output. If this number is less than the expected number (the number of tasks launched), the job state is set to `FAILED`.\r\n3. Code refactoring.\r\n\r\nSigned-off-by: Yinan Li <liyinan926@gmail.com>"},"comment":{"url":"https://api.github.com/repos/linkedin/gobblin/issues/comments/109157481","html_url":"https://github.com/linkedin/gobblin/pull/155#issuecomment-109157481","issue_url":"https://api.github.com/repos/linkedin/gobblin/issues/155","id":109157481,"user":{"login":"liyinan926","id":3709137,"avatar_url":"https://avatars.githubusercontent.com/u/3709137?v=3","gravatar_id":"","url":"https://api.github.com/users/liyinan926","html_url":"https://github.com/liyinan926","followers_url":"https://api.github.com/users/liyinan926/followers","following_url":"https://api.github.com/users/liyinan926/following{/other_user}","gists_url":"https://api.github.com/users/liyinan926/gists{/gist_id}","starred_url":"https://api.github.com/users/liyinan926/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liyinan926/subscriptions","organizations_url":"https://api.github.com/users/liyinan926/orgs","repos_url":"https://api.github.com/users/liyinan926/repos","events_url":"https://api.github.com/users/liyinan926/events{/privacy}","received_events_url":"https://api.github.com/users/liyinan926/received_events","type":"User","site_admin":false},"created_at":"2015-06-05T04:21:26Z","updated_at":"2015-06-05T04:21:26Z","body":"It all depends on which `JobCommitPolicy` is used. If `JobCommitPolicy.COMMIT_ON_FULL_SUCCESS` is used, the job state of a failed job won't be persisted since commit will not happen. For a successful job, the job state will be persisted if the commit is successful. If `JobCommitPolicy.COMMIT_ON_PARTIAL_SUCCESS` is used, however, the job state will be persisted if commit is successful. "}}