{"action":"created","issue":{"url":"https://api.github.com/repos/linkedin/gobblin/issues/254","labels_url":"https://api.github.com/repos/linkedin/gobblin/issues/254/labels{/name}","comments_url":"https://api.github.com/repos/linkedin/gobblin/issues/254/comments","events_url":"https://api.github.com/repos/linkedin/gobblin/issues/254/events","html_url":"https://github.com/linkedin/gobblin/pull/254","id":98009068,"number":254,"title":"Persist state and commit data on a per-dataset basis","user":{"login":"liyinan926","id":3709137,"avatar_url":"https://avatars.githubusercontent.com/u/3709137?v=3","gravatar_id":"","url":"https://api.github.com/users/liyinan926","html_url":"https://github.com/liyinan926","followers_url":"https://api.github.com/users/liyinan926/followers","following_url":"https://api.github.com/users/liyinan926/following{/other_user}","gists_url":"https://api.github.com/users/liyinan926/gists{/gist_id}","starred_url":"https://api.github.com/users/liyinan926/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liyinan926/subscriptions","organizations_url":"https://api.github.com/users/liyinan926/orgs","repos_url":"https://api.github.com/users/liyinan926/repos","events_url":"https://api.github.com/users/liyinan926/events{/privacy}","received_events_url":"https://api.github.com/users/liyinan926/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":17,"created_at":"2015-07-29T19:15:21Z","updated_at":"2015-08-03T05:27:22Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/linkedin/gobblin/pulls/254","html_url":"https://github.com/linkedin/gobblin/pull/254","diff_url":"https://github.com/linkedin/gobblin/pull/254.diff","patch_url":"https://github.com/linkedin/gobblin/pull/254.patch"},"body":"1. A `WorkUnit` can be specified to belong to a dataset group by using a new config property `dataset.urn`.\r\n2. Job/task state will be persisted on a per-dataset basis into different state store files, one per dataset. All `current` dataset state store files will be loaded in the next job run. \r\n3. Commit of data will be done on a per-dataset basis, allowing datasets to be committed independently.\r\n4. For existing job that does not have a dataset concept or only deals with a single dataset, the logic of job/task state persistence and data commit remains the same.\r\n \r\nSigned-off-by: Yinan Li <liyinan926@gmail.com>"},"comment":{"url":"https://api.github.com/repos/linkedin/gobblin/issues/comments/127128910","html_url":"https://github.com/linkedin/gobblin/pull/254#issuecomment-127128910","issue_url":"https://api.github.com/repos/linkedin/gobblin/issues/254","id":127128910,"user":{"login":"liyinan926","id":3709137,"avatar_url":"https://avatars.githubusercontent.com/u/3709137?v=3","gravatar_id":"","url":"https://api.github.com/users/liyinan926","html_url":"https://github.com/liyinan926","followers_url":"https://api.github.com/users/liyinan926/followers","following_url":"https://api.github.com/users/liyinan926/following{/other_user}","gists_url":"https://api.github.com/users/liyinan926/gists{/gist_id}","starred_url":"https://api.github.com/users/liyinan926/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liyinan926/subscriptions","organizations_url":"https://api.github.com/users/liyinan926/orgs","repos_url":"https://api.github.com/users/liyinan926/repos","events_url":"https://api.github.com/users/liyinan926/events{/privacy}","received_events_url":"https://api.github.com/users/liyinan926/received_events","type":"User","site_admin":false},"created_at":"2015-08-03T05:27:22Z","updated_at":"2015-08-03T05:27:22Z","body":"A `dataset.urn` is simply a more general form of dataset identifiers than an `Extract`, which feels to me too closely tight to table-based datasets. I thought about using `Extract` for dataset identifiers, but decided to use something more general in the end. I also think `Extract` itself is not super clear and intuitive. "}}