{"action":"created","comment":{"url":"https://api.github.com/repos/stripe/brushfire/pulls/comments/48160647","id":48160647,"diff_hunk":"@@ -3,81 +3,176 @@ package com.stripe.brushfire.local\n import com.stripe.brushfire._\n import com.twitter.algebird._\n \n-case class Trainer[K: Ordering, V, T: Monoid](\n-    trainingData: Iterable[Instance[K, V, T]],\n-    sampler: Sampler[K],\n-    trees: List[Tree[K, V, T]]) {\n \n-  private def updateTrees(fn: (Tree[K, V, T], Int, Map[LeafNode[K, V, T, Unit], Iterable[Instance[K, V, T]]]) => Tree[K, V, T]): Trainer[K, V, T] = {\n-    val newTrees = trees.zipWithIndex.par.map {\n-      case (tree, index) =>\n-        val byLeaf =\n-          trainingData.flatMap { instance =>\n-            val repeats = sampler.timesInTrainingSet(instance.id, instance.timestamp, index)\n-            if (repeats > 0) {\n-              tree.leafFor(instance.features).map { leaf =>\n-                1.to(repeats).toList.map { i => (instance, leaf) }\n-              }.getOrElse(Nil)\n-            } else {\n-              Nil\n-            }\n-          }.groupBy { _._2 }\n-            .mapValues { _.map { _._1 } }\n-        fn(tree, index, byLeaf)\n-    }\n-    copy(trees = newTrees.toList)\n+trait TrainingStep[K,V,T] {\n+\n+  type K1\n+  type V1","path":"brushfire-core/src/main/scala/com/stripe/brushfire/local/Trainer.scala","position":30,"original_position":30,"commit_id":"5bcace3d6cfec71321d84c6b627df1b59bd8de24","original_commit_id":"5bcace3d6cfec71321d84c6b627df1b59bd8de24","user":{"login":"tixxit","id":120314,"avatar_url":"https://avatars.githubusercontent.com/u/120314?v=3","gravatar_id":"","url":"https://api.github.com/users/tixxit","html_url":"https://github.com/tixxit","followers_url":"https://api.github.com/users/tixxit/followers","following_url":"https://api.github.com/users/tixxit/following{/other_user}","gists_url":"https://api.github.com/users/tixxit/gists{/gist_id}","starred_url":"https://api.github.com/users/tixxit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tixxit/subscriptions","organizations_url":"https://api.github.com/users/tixxit/orgs","repos_url":"https://api.github.com/users/tixxit/repos","events_url":"https://api.github.com/users/tixxit/events{/privacy}","received_events_url":"https://api.github.com/users/tixxit/received_events","type":"User","site_admin":false},"body":"This is a per-leaf (per K1) value that is generated by prepare, based on the instance. We then sum up each value per leaf (per key). This final value is the one given to `lift`.","created_at":"2015-12-21T16:13:45Z","updated_at":"2015-12-21T16:13:45Z","html_url":"https://github.com/stripe/brushfire/pull/77#discussion_r48160647","pull_request_url":"https://api.github.com/repos/stripe/brushfire/pulls/77","_links":{"self":{"href":"https://api.github.com/repos/stripe/brushfire/pulls/comments/48160647"},"html":{"href":"https://github.com/stripe/brushfire/pull/77#discussion_r48160647"},"pull_request":{"href":"https://api.github.com/repos/stripe/brushfire/pulls/77"}}},"pull_request":{"url":"https://api.github.com/repos/stripe/brushfire/pulls/77","id":54228930,"html_url":"https://github.com/stripe/brushfire/pull/77","diff_url":"https://github.com/stripe/brushfire/pull/77.diff","patch_url":"https://github.com/stripe/brushfire/pull/77.patch","issue_url":"https://api.github.com/repos/stripe/brushfire/issues/77","number":77,"state":"open","locked":false,"title":"WIP: TrainingStep","user":{"login":"avibryant","id":9892,"avatar_url":"https://avatars.githubusercontent.com/u/9892?v=3","gravatar_id":"","url":"https://api.github.com/users/avibryant","html_url":"https://github.com/avibryant","followers_url":"https://api.github.com/users/avibryant/followers","following_url":"https://api.github.com/users/avibryant/following{/other_user}","gists_url":"https://api.github.com/users/avibryant/gists{/gist_id}","starred_url":"https://api.github.com/users/avibryant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/avibryant/subscriptions","organizations_url":"https://api.github.com/users/avibryant/orgs","repos_url":"https://api.github.com/users/avibryant/repos","events_url":"https://api.github.com/users/avibryant/events{/privacy}","received_events_url":"https://api.github.com/users/avibryant/received_events","type":"User","site_admin":false},"body":"attn @tixxit @non\r\n\r\nThe intent here is to capture the basic mechanics of various training steps - updateTargets, expand, prune, etc - in a way that can be reused in multiple execution environments (local, scalding, spark, ...).\r\n\r\nFor now, this is all added directly to and used only by the Local trainer. The near-term impact is that the local trainer will stream over its input data in the same way that the distributed trainers do, rather than requiring it to all be loaded into memory. For this PR to be complete, we should move the training steps to their own module (maybe also doing https://github.com/stripe/brushfire/issues/51), and refactor the scalding trainer to use them.\r\n\r\nIt's very possible that this is too much or too little abstraction - right now it seems a bit overfit to the needs of the specific training steps and platforms we support, and I suspect it will be brittle going forward. (In fact, featureImportance already doesn't work with this, though I can argue that we should move to a TreeTraversal-based strategy for that which would). At the same time, I think *some* approach like this will be valuable going forward, and I think it's better to start going imperfectly down this path.\r\n","created_at":"2015-12-21T07:51:29Z","updated_at":"2015-12-21T16:13:45Z","closed_at":null,"merged_at":null,"merge_commit_sha":"c9a261c5d055cb75da612a65d2bc032c44c59415","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/stripe/brushfire/pulls/77/commits","review_comments_url":"https://api.github.com/repos/stripe/brushfire/pulls/77/comments","review_comment_url":"https://api.github.com/repos/stripe/brushfire/pulls/comments{/number}","comments_url":"https://api.github.com/repos/stripe/brushfire/issues/77/comments","statuses_url":"https://api.github.com/repos/stripe/brushfire/statuses/5bcace3d6cfec71321d84c6b627df1b59bd8de24","head":{"label":"stripe:avi-training-step","ref":"avi-training-step","sha":"5bcace3d6cfec71321d84c6b627df1b59bd8de24","user":{"login":"stripe","id":856813,"avatar_url":"https://avatars.githubusercontent.com/u/856813?v=3","gravatar_id":"","url":"https://api.github.com/users/stripe","html_url":"https://github.com/stripe","followers_url":"https://api.github.com/users/stripe/followers","following_url":"https://api.github.com/users/stripe/following{/other_user}","gists_url":"https://api.github.com/users/stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stripe/subscriptions","organizations_url":"https://api.github.com/users/stripe/orgs","repos_url":"https://api.github.com/users/stripe/repos","events_url":"https://api.github.com/users/stripe/events{/privacy}","received_events_url":"https://api.github.com/users/stripe/received_events","type":"Organization","site_admin":false},"repo":{"id":26926567,"name":"brushfire","full_name":"stripe/brushfire","owner":{"login":"stripe","id":856813,"avatar_url":"https://avatars.githubusercontent.com/u/856813?v=3","gravatar_id":"","url":"https://api.github.com/users/stripe","html_url":"https://github.com/stripe","followers_url":"https://api.github.com/users/stripe/followers","following_url":"https://api.github.com/users/stripe/following{/other_user}","gists_url":"https://api.github.com/users/stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stripe/subscriptions","organizations_url":"https://api.github.com/users/stripe/orgs","repos_url":"https://api.github.com/users/stripe/repos","events_url":"https://api.github.com/users/stripe/events{/privacy}","received_events_url":"https://api.github.com/users/stripe/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/stripe/brushfire","description":"Distributed decision tree ensemble learning in Scala","fork":false,"url":"https://api.github.com/repos/stripe/brushfire","forks_url":"https://api.github.com/repos/stripe/brushfire/forks","keys_url":"https://api.github.com/repos/stripe/brushfire/keys{/key_id}","collaborators_url":"https://api.github.com/repos/stripe/brushfire/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/stripe/brushfire/teams","hooks_url":"https://api.github.com/repos/stripe/brushfire/hooks","issue_events_url":"https://api.github.com/repos/stripe/brushfire/issues/events{/number}","events_url":"https://api.github.com/repos/stripe/brushfire/events","assignees_url":"https://api.github.com/repos/stripe/brushfire/assignees{/user}","branches_url":"https://api.github.com/repos/stripe/brushfire/branches{/branch}","tags_url":"https://api.github.com/repos/stripe/brushfire/tags","blobs_url":"https://api.github.com/repos/stripe/brushfire/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/stripe/brushfire/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/stripe/brushfire/git/refs{/sha}","trees_url":"https://api.github.com/repos/stripe/brushfire/git/trees{/sha}","statuses_url":"https://api.github.com/repos/stripe/brushfire/statuses/{sha}","languages_url":"https://api.github.com/repos/stripe/brushfire/languages","stargazers_url":"https://api.github.com/repos/stripe/brushfire/stargazers","contributors_url":"https://api.github.com/repos/stripe/brushfire/contributors","subscribers_url":"https://api.github.com/repos/stripe/brushfire/subscribers","subscription_url":"https://api.github.com/repos/stripe/brushfire/subscription","commits_url":"https://api.github.com/repos/stripe/brushfire/commits{/sha}","git_commits_url":"https://api.github.com/repos/stripe/brushfire/git/commits{/sha}","comments_url":"https://api.github.com/repos/stripe/brushfire/comments{/number}","issue_comment_url":"https://api.github.com/repos/stripe/brushfire/issues/comments{/number}","contents_url":"https://api.github.com/repos/stripe/brushfire/contents/{+path}","compare_url":"https://api.github.com/repos/stripe/brushfire/compare/{base}...{head}","merges_url":"https://api.github.com/repos/stripe/brushfire/merges","archive_url":"https://api.github.com/repos/stripe/brushfire/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/stripe/brushfire/downloads","issues_url":"https://api.github.com/repos/stripe/brushfire/issues{/number}","pulls_url":"https://api.github.com/repos/stripe/brushfire/pulls{/number}","milestones_url":"https://api.github.com/repos/stripe/brushfire/milestones{/number}","notifications_url":"https://api.github.com/repos/stripe/brushfire/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/stripe/brushfire/labels{/name}","releases_url":"https://api.github.com/repos/stripe/brushfire/releases{/id}","created_at":"2014-11-20T18:47:07Z","updated_at":"2015-12-18T07:54:06Z","pushed_at":"2015-12-21T07:51:29Z","git_url":"git://github.com/stripe/brushfire.git","ssh_url":"git@github.com:stripe/brushfire.git","clone_url":"https://github.com/stripe/brushfire.git","svn_url":"https://github.com/stripe/brushfire","homepage":null,"size":1345,"stargazers_count":282,"watchers_count":282,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":27,"mirror_url":null,"open_issues_count":26,"forks":27,"open_issues":26,"watchers":282,"default_branch":"master"}},"base":{"label":"stripe:master","ref":"master","sha":"79937acd822a66213aaf5ae8deb3af3297bfab51","user":{"login":"stripe","id":856813,"avatar_url":"https://avatars.githubusercontent.com/u/856813?v=3","gravatar_id":"","url":"https://api.github.com/users/stripe","html_url":"https://github.com/stripe","followers_url":"https://api.github.com/users/stripe/followers","following_url":"https://api.github.com/users/stripe/following{/other_user}","gists_url":"https://api.github.com/users/stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stripe/subscriptions","organizations_url":"https://api.github.com/users/stripe/orgs","repos_url":"https://api.github.com/users/stripe/repos","events_url":"https://api.github.com/users/stripe/events{/privacy}","received_events_url":"https://api.github.com/users/stripe/received_events","type":"Organization","site_admin":false},"repo":{"id":26926567,"name":"brushfire","full_name":"stripe/brushfire","owner":{"login":"stripe","id":856813,"avatar_url":"https://avatars.githubusercontent.com/u/856813?v=3","gravatar_id":"","url":"https://api.github.com/users/stripe","html_url":"https://github.com/stripe","followers_url":"https://api.github.com/users/stripe/followers","following_url":"https://api.github.com/users/stripe/following{/other_user}","gists_url":"https://api.github.com/users/stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stripe/subscriptions","organizations_url":"https://api.github.com/users/stripe/orgs","repos_url":"https://api.github.com/users/stripe/repos","events_url":"https://api.github.com/users/stripe/events{/privacy}","received_events_url":"https://api.github.com/users/stripe/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/stripe/brushfire","description":"Distributed decision tree ensemble learning in Scala","fork":false,"url":"https://api.github.com/repos/stripe/brushfire","forks_url":"https://api.github.com/repos/stripe/brushfire/forks","keys_url":"https://api.github.com/repos/stripe/brushfire/keys{/key_id}","collaborators_url":"https://api.github.com/repos/stripe/brushfire/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/stripe/brushfire/teams","hooks_url":"https://api.github.com/repos/stripe/brushfire/hooks","issue_events_url":"https://api.github.com/repos/stripe/brushfire/issues/events{/number}","events_url":"https://api.github.com/repos/stripe/brushfire/events","assignees_url":"https://api.github.com/repos/stripe/brushfire/assignees{/user}","branches_url":"https://api.github.com/repos/stripe/brushfire/branches{/branch}","tags_url":"https://api.github.com/repos/stripe/brushfire/tags","blobs_url":"https://api.github.com/repos/stripe/brushfire/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/stripe/brushfire/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/stripe/brushfire/git/refs{/sha}","trees_url":"https://api.github.com/repos/stripe/brushfire/git/trees{/sha}","statuses_url":"https://api.github.com/repos/stripe/brushfire/statuses/{sha}","languages_url":"https://api.github.com/repos/stripe/brushfire/languages","stargazers_url":"https://api.github.com/repos/stripe/brushfire/stargazers","contributors_url":"https://api.github.com/repos/stripe/brushfire/contributors","subscribers_url":"https://api.github.com/repos/stripe/brushfire/subscribers","subscription_url":"https://api.github.com/repos/stripe/brushfire/subscription","commits_url":"https://api.github.com/repos/stripe/brushfire/commits{/sha}","git_commits_url":"https://api.github.com/repos/stripe/brushfire/git/commits{/sha}","comments_url":"https://api.github.com/repos/stripe/brushfire/comments{/number}","issue_comment_url":"https://api.github.com/repos/stripe/brushfire/issues/comments{/number}","contents_url":"https://api.github.com/repos/stripe/brushfire/contents/{+path}","compare_url":"https://api.github.com/repos/stripe/brushfire/compare/{base}...{head}","merges_url":"https://api.github.com/repos/stripe/brushfire/merges","archive_url":"https://api.github.com/repos/stripe/brushfire/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/stripe/brushfire/downloads","issues_url":"https://api.github.com/repos/stripe/brushfire/issues{/number}","pulls_url":"https://api.github.com/repos/stripe/brushfire/pulls{/number}","milestones_url":"https://api.github.com/repos/stripe/brushfire/milestones{/number}","notifications_url":"https://api.github.com/repos/stripe/brushfire/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/stripe/brushfire/labels{/name}","releases_url":"https://api.github.com/repos/stripe/brushfire/releases{/id}","created_at":"2014-11-20T18:47:07Z","updated_at":"2015-12-18T07:54:06Z","pushed_at":"2015-12-21T07:51:29Z","git_url":"git://github.com/stripe/brushfire.git","ssh_url":"git@github.com:stripe/brushfire.git","clone_url":"https://github.com/stripe/brushfire.git","svn_url":"https://github.com/stripe/brushfire","homepage":null,"size":1345,"stargazers_count":282,"watchers_count":282,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":27,"mirror_url":null,"open_issues_count":26,"forks":27,"open_issues":26,"watchers":282,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/stripe/brushfire/pulls/77"},"html":{"href":"https://github.com/stripe/brushfire/pull/77"},"issue":{"href":"https://api.github.com/repos/stripe/brushfire/issues/77"},"comments":{"href":"https://api.github.com/repos/stripe/brushfire/issues/77/comments"},"review_comments":{"href":"https://api.github.com/repos/stripe/brushfire/pulls/77/comments"},"review_comment":{"href":"https://api.github.com/repos/stripe/brushfire/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/stripe/brushfire/pulls/77/commits"},"statuses":{"href":"https://api.github.com/repos/stripe/brushfire/statuses/5bcace3d6cfec71321d84c6b627df1b59bd8de24"}}}}