{"action":"created","comment":{"url":"https://api.github.com/repos/thieman/dagobah/pulls/comments/37493834","id":37493834,"diff_hunk":"@@ -467,12 +469,119 @@ def initialize_snapshot(self):\n         if not is_valid:\n             raise DagobahError(reason)\n \n-        self.snapshot = snapshot_to_validate\n+        verified = self.verify()\n+\n+        if not verified:\n+            raise DagobahError(\"Job has a cycle, possibly within another Job \"\n+                               + \"reference\")\n+\n+        # Due to thread locks in underlying tasks, they must be manually copied\n+        tasks_copy = {}\n+        for name, task in self.tasks.iteritems():\n+            if self.implements_runnable(task):\n+                tasks_copy[name] = Task(self, task.command, task.name,\n+                                        soft_timeout=task.soft_timeout,\n+                                        hard_timeout=task.hard_timeout,\n+                                        hostname=task.hostname)\n+            elif self.implements_expandable(task):\n+                tasks_copy[name] = JobTask(self, task.name,\n+                                           task.target_job_name)\n+            else:\n+                raise DagobahError(\"Malformed task neither a Task nor JobTask\")","path":"dagobah/core/job.py","position":85,"original_position":85,"commit_id":"93e3baebf7ec8e5779e214b03463d1a41c88068f","original_commit_id":"93e3baebf7ec8e5779e214b03463d1a41c88068f","user":{"login":"rclough","id":186589,"avatar_url":"https://avatars.githubusercontent.com/u/186589?v=3","gravatar_id":"","url":"https://api.github.com/users/rclough","html_url":"https://github.com/rclough","followers_url":"https://api.github.com/users/rclough/followers","following_url":"https://api.github.com/users/rclough/following{/other_user}","gists_url":"https://api.github.com/users/rclough/gists{/gist_id}","starred_url":"https://api.github.com/users/rclough/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rclough/subscriptions","organizations_url":"https://api.github.com/users/rclough/orgs","repos_url":"https://api.github.com/users/rclough/repos","events_url":"https://api.github.com/users/rclough/events{/privacy}","received_events_url":"https://api.github.com/users/rclough/received_events","type":"User","site_admin":false},"body":"end up needing to do what exactly?\r\n\r\nI mainly wrote the else part as a catch-all/defensive programming, but I generally never expect the code to hit that line.","created_at":"2015-08-20T04:19:02Z","updated_at":"2015-08-20T04:19:02Z","html_url":"https://github.com/thieman/dagobah/pull/162#discussion_r37493834","pull_request_url":"https://api.github.com/repos/thieman/dagobah/pulls/162","_links":{"self":{"href":"https://api.github.com/repos/thieman/dagobah/pulls/comments/37493834"},"html":{"href":"https://github.com/thieman/dagobah/pull/162#discussion_r37493834"},"pull_request":{"href":"https://api.github.com/repos/thieman/dagobah/pulls/162"}}},"pull_request":{"url":"https://api.github.com/repos/thieman/dagobah/pulls/162","id":42756184,"html_url":"https://github.com/thieman/dagobah/pull/162","diff_url":"https://github.com/thieman/dagobah/pull/162.diff","patch_url":"https://github.com/thieman/dagobah/pull/162.patch","issue_url":"https://api.github.com/repos/thieman/dagobah/issues/162","number":162,"state":"open","locked":false,"title":"Rmc dev job in job expansion","user":{"login":"rclough","id":186589,"avatar_url":"https://avatars.githubusercontent.com/u/186589?v=3","gravatar_id":"","url":"https://api.github.com/users/rclough","html_url":"https://github.com/rclough","followers_url":"https://api.github.com/users/rclough/followers","following_url":"https://api.github.com/users/rclough/following{/other_user}","gists_url":"https://api.github.com/users/rclough/gists{/gist_id}","starred_url":"https://api.github.com/users/rclough/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rclough/subscriptions","organizations_url":"https://api.github.com/users/rclough/orgs","repos_url":"https://api.github.com/users/rclough/repos","events_url":"https://api.github.com/users/rclough/events{/privacy}","received_events_url":"https://api.github.com/users/rclough/received_events","type":"User","site_admin":false},"body":"~~This isn't a final PR, just the introduction of job expansion for scrutiny and debugging.\r\n\r\n\r\nRight now, I've manually tested a number of dags and verified that job expansion works as we would like it. However, there are some side effects of the new implementation that are causing issues that are causing other tests to fail. I wanted to get you opinion on what's going on.\r\n\r\nHere's my general understanding:\r\n\r\n* In order to get job expansion to work, not only do we have to create a snapshot of the DAG graph, but we also have to create a snapshot of the task list, as dagobah uses both of these to run a job, so expansion requires the inheretied jobs to be in both maps\r\n* In order to make a snapshot, I'm doing a `deepcopy(self.tasks)`\r\n* There are apparently some thread locks within the tasks that, upon deepcopy, are causing it to barf (because you can't copy thread locks)\r\n\r\nI was wondering what you think would be the best solution for this?\r\n\r\nMy heads a bit fuzzy from debugging all day, but my suspicion is I'll have manually copy by looping over the `tasks` map, and populating a new snapshot map with new task objects, generated from the deserialized json of the current `tasks` map.\r\n\r\nBut perhaps there is a simpler solution? I'm not very experienced with python threading.~~\r\n\r\n\r\nThis has now been figured out, cleaned, and is ready to be a full PR more or less. Explanation to come.","created_at":"2015-08-18T22:21:19Z","updated_at":"2015-08-20T04:19:02Z","closed_at":null,"merged_at":null,"merge_commit_sha":"7bba5ad9e77cdda60899df18fe32dc0991d503af","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/thieman/dagobah/pulls/162/commits","review_comments_url":"https://api.github.com/repos/thieman/dagobah/pulls/162/comments","review_comment_url":"https://api.github.com/repos/thieman/dagobah/pulls/comments{/number}","comments_url":"https://api.github.com/repos/thieman/dagobah/issues/162/comments","statuses_url":"https://api.github.com/repos/thieman/dagobah/statuses/93e3baebf7ec8e5779e214b03463d1a41c88068f","head":{"label":"rclough:rmc-dev-job-in-job-expansion","ref":"rmc-dev-job-in-job-expansion","sha":"93e3baebf7ec8e5779e214b03463d1a41c88068f","user":{"login":"rclough","id":186589,"avatar_url":"https://avatars.githubusercontent.com/u/186589?v=3","gravatar_id":"","url":"https://api.github.com/users/rclough","html_url":"https://github.com/rclough","followers_url":"https://api.github.com/users/rclough/followers","following_url":"https://api.github.com/users/rclough/following{/other_user}","gists_url":"https://api.github.com/users/rclough/gists{/gist_id}","starred_url":"https://api.github.com/users/rclough/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rclough/subscriptions","organizations_url":"https://api.github.com/users/rclough/orgs","repos_url":"https://api.github.com/users/rclough/repos","events_url":"https://api.github.com/users/rclough/events{/privacy}","received_events_url":"https://api.github.com/users/rclough/received_events","type":"User","site_admin":false},"repo":{"id":19712189,"name":"dagobah","full_name":"rclough/dagobah","owner":{"login":"rclough","id":186589,"avatar_url":"https://avatars.githubusercontent.com/u/186589?v=3","gravatar_id":"","url":"https://api.github.com/users/rclough","html_url":"https://github.com/rclough","followers_url":"https://api.github.com/users/rclough/followers","following_url":"https://api.github.com/users/rclough/following{/other_user}","gists_url":"https://api.github.com/users/rclough/gists{/gist_id}","starred_url":"https://api.github.com/users/rclough/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rclough/subscriptions","organizations_url":"https://api.github.com/users/rclough/orgs","repos_url":"https://api.github.com/users/rclough/repos","events_url":"https://api.github.com/users/rclough/events{/privacy}","received_events_url":"https://api.github.com/users/rclough/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/rclough/dagobah","description":"Simple DAG-based job scheduler in Python","fork":true,"url":"https://api.github.com/repos/rclough/dagobah","forks_url":"https://api.github.com/repos/rclough/dagobah/forks","keys_url":"https://api.github.com/repos/rclough/dagobah/keys{/key_id}","collaborators_url":"https://api.github.com/repos/rclough/dagobah/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/rclough/dagobah/teams","hooks_url":"https://api.github.com/repos/rclough/dagobah/hooks","issue_events_url":"https://api.github.com/repos/rclough/dagobah/issues/events{/number}","events_url":"https://api.github.com/repos/rclough/dagobah/events","assignees_url":"https://api.github.com/repos/rclough/dagobah/assignees{/user}","branches_url":"https://api.github.com/repos/rclough/dagobah/branches{/branch}","tags_url":"https://api.github.com/repos/rclough/dagobah/tags","blobs_url":"https://api.github.com/repos/rclough/dagobah/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/rclough/dagobah/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/rclough/dagobah/git/refs{/sha}","trees_url":"https://api.github.com/repos/rclough/dagobah/git/trees{/sha}","statuses_url":"https://api.github.com/repos/rclough/dagobah/statuses/{sha}","languages_url":"https://api.github.com/repos/rclough/dagobah/languages","stargazers_url":"https://api.github.com/repos/rclough/dagobah/stargazers","contributors_url":"https://api.github.com/repos/rclough/dagobah/contributors","subscribers_url":"https://api.github.com/repos/rclough/dagobah/subscribers","subscription_url":"https://api.github.com/repos/rclough/dagobah/subscription","commits_url":"https://api.github.com/repos/rclough/dagobah/commits{/sha}","git_commits_url":"https://api.github.com/repos/rclough/dagobah/git/commits{/sha}","comments_url":"https://api.github.com/repos/rclough/dagobah/comments{/number}","issue_comment_url":"https://api.github.com/repos/rclough/dagobah/issues/comments{/number}","contents_url":"https://api.github.com/repos/rclough/dagobah/contents/{+path}","compare_url":"https://api.github.com/repos/rclough/dagobah/compare/{base}...{head}","merges_url":"https://api.github.com/repos/rclough/dagobah/merges","archive_url":"https://api.github.com/repos/rclough/dagobah/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/rclough/dagobah/downloads","issues_url":"https://api.github.com/repos/rclough/dagobah/issues{/number}","pulls_url":"https://api.github.com/repos/rclough/dagobah/pulls{/number}","milestones_url":"https://api.github.com/repos/rclough/dagobah/milestones{/number}","notifications_url":"https://api.github.com/repos/rclough/dagobah/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/rclough/dagobah/labels{/name}","releases_url":"https://api.github.com/repos/rclough/dagobah/releases{/id}","created_at":"2014-05-12T19:28:19Z","updated_at":"2014-12-01T23:27:49Z","pushed_at":"2015-08-19T19:02:04Z","git_url":"git://github.com/rclough/dagobah.git","ssh_url":"git@github.com:rclough/dagobah.git","clone_url":"https://github.com/rclough/dagobah.git","svn_url":"https://github.com/rclough/dagobah","homepage":null,"size":2647,"stargazers_count":0,"watchers_count":0,"language":"CSS","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"thieman:dev-job-in-job","ref":"dev-job-in-job","sha":"2192f21f45cc39cf7a54f012ea474e47dfe2329e","user":{"login":"thieman","id":1414305,"avatar_url":"https://avatars.githubusercontent.com/u/1414305?v=3","gravatar_id":"","url":"https://api.github.com/users/thieman","html_url":"https://github.com/thieman","followers_url":"https://api.github.com/users/thieman/followers","following_url":"https://api.github.com/users/thieman/following{/other_user}","gists_url":"https://api.github.com/users/thieman/gists{/gist_id}","starred_url":"https://api.github.com/users/thieman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thieman/subscriptions","organizations_url":"https://api.github.com/users/thieman/orgs","repos_url":"https://api.github.com/users/thieman/repos","events_url":"https://api.github.com/users/thieman/events{/privacy}","received_events_url":"https://api.github.com/users/thieman/received_events","type":"User","site_admin":false},"repo":{"id":9983531,"name":"dagobah","full_name":"thieman/dagobah","owner":{"login":"thieman","id":1414305,"avatar_url":"https://avatars.githubusercontent.com/u/1414305?v=3","gravatar_id":"","url":"https://api.github.com/users/thieman","html_url":"https://github.com/thieman","followers_url":"https://api.github.com/users/thieman/followers","following_url":"https://api.github.com/users/thieman/following{/other_user}","gists_url":"https://api.github.com/users/thieman/gists{/gist_id}","starred_url":"https://api.github.com/users/thieman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thieman/subscriptions","organizations_url":"https://api.github.com/users/thieman/orgs","repos_url":"https://api.github.com/users/thieman/repos","events_url":"https://api.github.com/users/thieman/events{/privacy}","received_events_url":"https://api.github.com/users/thieman/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/thieman/dagobah","description":"Simple DAG-based job scheduler in Python","fork":false,"url":"https://api.github.com/repos/thieman/dagobah","forks_url":"https://api.github.com/repos/thieman/dagobah/forks","keys_url":"https://api.github.com/repos/thieman/dagobah/keys{/key_id}","collaborators_url":"https://api.github.com/repos/thieman/dagobah/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/thieman/dagobah/teams","hooks_url":"https://api.github.com/repos/thieman/dagobah/hooks","issue_events_url":"https://api.github.com/repos/thieman/dagobah/issues/events{/number}","events_url":"https://api.github.com/repos/thieman/dagobah/events","assignees_url":"https://api.github.com/repos/thieman/dagobah/assignees{/user}","branches_url":"https://api.github.com/repos/thieman/dagobah/branches{/branch}","tags_url":"https://api.github.com/repos/thieman/dagobah/tags","blobs_url":"https://api.github.com/repos/thieman/dagobah/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/thieman/dagobah/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/thieman/dagobah/git/refs{/sha}","trees_url":"https://api.github.com/repos/thieman/dagobah/git/trees{/sha}","statuses_url":"https://api.github.com/repos/thieman/dagobah/statuses/{sha}","languages_url":"https://api.github.com/repos/thieman/dagobah/languages","stargazers_url":"https://api.github.com/repos/thieman/dagobah/stargazers","contributors_url":"https://api.github.com/repos/thieman/dagobah/contributors","subscribers_url":"https://api.github.com/repos/thieman/dagobah/subscribers","subscription_url":"https://api.github.com/repos/thieman/dagobah/subscription","commits_url":"https://api.github.com/repos/thieman/dagobah/commits{/sha}","git_commits_url":"https://api.github.com/repos/thieman/dagobah/git/commits{/sha}","comments_url":"https://api.github.com/repos/thieman/dagobah/comments{/number}","issue_comment_url":"https://api.github.com/repos/thieman/dagobah/issues/comments{/number}","contents_url":"https://api.github.com/repos/thieman/dagobah/contents/{+path}","compare_url":"https://api.github.com/repos/thieman/dagobah/compare/{base}...{head}","merges_url":"https://api.github.com/repos/thieman/dagobah/merges","archive_url":"https://api.github.com/repos/thieman/dagobah/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/thieman/dagobah/downloads","issues_url":"https://api.github.com/repos/thieman/dagobah/issues{/number}","pulls_url":"https://api.github.com/repos/thieman/dagobah/pulls{/number}","milestones_url":"https://api.github.com/repos/thieman/dagobah/milestones{/number}","notifications_url":"https://api.github.com/repos/thieman/dagobah/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/thieman/dagobah/labels{/name}","releases_url":"https://api.github.com/repos/thieman/dagobah/releases{/id}","created_at":"2013-05-10T14:48:40Z","updated_at":"2015-08-18T17:36:11Z","pushed_at":"2015-08-19T19:02:05Z","git_url":"git://github.com/thieman/dagobah.git","ssh_url":"git@github.com:thieman/dagobah.git","clone_url":"https://github.com/thieman/dagobah.git","svn_url":"https://github.com/thieman/dagobah","homepage":null,"size":3837,"stargazers_count":288,"watchers_count":288,"language":"CSS","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":57,"mirror_url":null,"open_issues_count":45,"forks":57,"open_issues":45,"watchers":288,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/thieman/dagobah/pulls/162"},"html":{"href":"https://github.com/thieman/dagobah/pull/162"},"issue":{"href":"https://api.github.com/repos/thieman/dagobah/issues/162"},"comments":{"href":"https://api.github.com/repos/thieman/dagobah/issues/162/comments"},"review_comments":{"href":"https://api.github.com/repos/thieman/dagobah/pulls/162/comments"},"review_comment":{"href":"https://api.github.com/repos/thieman/dagobah/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/thieman/dagobah/pulls/162/commits"},"statuses":{"href":"https://api.github.com/repos/thieman/dagobah/statuses/93e3baebf7ec8e5779e214b03463d1a41c88068f"}}}}