{"action":"created","issue":{"url":"https://api.github.com/repos/amplab/keystone/issues/142","labels_url":"https://api.github.com/repos/amplab/keystone/issues/142/labels{/name}","comments_url":"https://api.github.com/repos/amplab/keystone/issues/142/comments","events_url":"https://api.github.com/repos/amplab/keystone/issues/142/events","html_url":"https://github.com/amplab/keystone/pull/142","id":85996363,"number":142,"title":"Dag construction","user":{"login":"tomerk","id":676357,"avatar_url":"https://avatars.githubusercontent.com/u/676357?v=3","gravatar_id":"","url":"https://api.github.com/users/tomerk","html_url":"https://github.com/tomerk","followers_url":"https://api.github.com/users/tomerk/followers","following_url":"https://api.github.com/users/tomerk/following{/other_user}","gists_url":"https://api.github.com/users/tomerk/gists{/gist_id}","starred_url":"https://api.github.com/users/tomerk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomerk/subscriptions","organizations_url":"https://api.github.com/users/tomerk/orgs","repos_url":"https://api.github.com/users/tomerk/repos","events_url":"https://api.github.com/users/tomerk/events{/privacy}","received_events_url":"https://api.github.com/users/tomerk/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-06-08T02:36:52Z","updated_at":"2015-06-14T02:59:27Z","closed_at":"2015-06-14T02:58:53Z","pull_request":{"url":"https://api.github.com/repos/amplab/keystone/pulls/142","html_url":"https://github.com/amplab/keystone/pull/142","diff_url":"https://github.com/amplab/keystone/pull/142.diff","patch_url":"https://github.com/amplab/keystone/pull/142.patch"},"body":"@etrain @shivaram \r\nStill working on this, but I think it's ready to start getting feedback.\r\n\r\n## What is a Pipeline?\r\nA `Pipeline` is made up of a sequence of nodes (where adjacent nodes have matching output/input types). It exposes a ```.fit()``` that outputs a PipelineModel (a transformer made up of a chain of transformers, that applies each one on the input in sequence)\r\n\r\n`Pipeline.fit()` is roughly defined as follows:\r\n```scala\r\ndef fit(): PipelineModel[A, B] = {\r\n  var transformers: Seq[Transformer] = Seq()\r\n  nodes.foreach { node =>\r\n    val nextTransformer = node match {\r\n      case EstimatorWithData(est, data) => {\r\n        est.fit( PipelineModel(transformers).apply(data) )\r\n      }\r\n      case pipeline: Pipeline => pipeline.fit()\r\n      case transformer: Transformer => transformer\r\n    }\r\n    transformers = transformers :+ nextTransformer\r\n  }\r\n\r\n  PipelineModel(transformers)\r\n}\r\n```\r\nIt's got more to it, but that's the main idea. Start w/ an empty seq of Transformers. Go through your nodes one by one. If you see a transformer, add it to your transformers as-is. If you see a pipeline, fit it then add the output transformer. If you see an estimator w/ data, first apply your current seq of transformers to the data, then fit the estimator on the transformed data. Finally wrap your Seq of transformers in a PipelineModel (a transformer made up of a chain of transformers)\r\n\r\n## What is a Node?\r\nA ```Node``` is anything that can meaningfully be chained as part of a Pipeline (basically anything that ```Pipeline.fit()``` knows what to do with)\r\nTransformers are nodes\r\nA pipeline is a node\r\nThe scatter operator is a node\r\nEstimators on their own are NOT nodes, but Estimators w/ data attached are.\r\nLabeledEstimators w/ data attached are also nodes.\r\n\r\nNodes have a `rewrite` method (output a seq of nodes that don't change the fit output when appended onto an existing pipeline as opposed to just outputting the current node, but make the DAG flatter/more interpretable and in the case of scatter nodes avoids repeating work)\r\n\r\nNodes also have a `canSafelyPrependExtraNodes` method that returns true if it is safe for the Pipeline that directly encloses this node to have new nodes prepended. E.g. false for estimators w/ data because the attached data type must match the input type of the enclosing Pipeline, but true for Transformers. For scatter nodes it depends on what they internally contain. \r\n\r\n## What I want feedback on\r\nI'm not entirely happy with how leaky the abstractions are in `Pipeline.fit()` (creating new DAG operators or new types of Nodes requires updating Pipeline.fit() no matter what). Without a better idea of how @etrain wants to do checkpointing stuff I can't quite be sure how to proceed.\r\n\r\nI'm also not sure what the right balance is between calling `node.rewrite` everywhere versus having constructors call rewrite but not being able to take advantage of case classes in some cases. E.g. `Pipeline` has a constructor that forces a rewrite which means it can't be a case class, so I have to implement unapply, toString, etc. methods but on the other hand I don't have to put rewrite everywhere to make sure it ends up called. (technically we could get away with only ever calling rewrite right before a fit, but the DAG wouldn't be as clean & predictable when trying to drill down programmatically if we don't call rewrite when constructing or chaining things) \r\n\r\nSome of the method names are kind of ugly (e.g. canSafelyPrependExtraNodes), and if I end up keeping them, suggestions for alternative names are welcome.\r\n\r\n(misc note: most of the changed files are just changing imports to be Workflow nodes instead of previous nodes)"},"comment":{"url":"https://api.github.com/repos/amplab/keystone/issues/comments/111774402","html_url":"https://github.com/amplab/keystone/pull/142#issuecomment-111774402","issue_url":"https://api.github.com/repos/amplab/keystone/issues/142","id":111774402,"user":{"login":"tomerk","id":676357,"avatar_url":"https://avatars.githubusercontent.com/u/676357?v=3","gravatar_id":"","url":"https://api.github.com/users/tomerk","html_url":"https://github.com/tomerk","followers_url":"https://api.github.com/users/tomerk/followers","following_url":"https://api.github.com/users/tomerk/following{/other_user}","gists_url":"https://api.github.com/users/tomerk/gists{/gist_id}","starred_url":"https://api.github.com/users/tomerk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomerk/subscriptions","organizations_url":"https://api.github.com/users/tomerk/orgs","repos_url":"https://api.github.com/users/tomerk/repos","events_url":"https://api.github.com/users/tomerk/events{/privacy}","received_events_url":"https://api.github.com/users/tomerk/received_events","type":"User","site_admin":false},"created_at":"2015-06-14T02:59:27Z","updated_at":"2015-06-14T02:59:27Z","body":"Future work happening on my branch dag-construction-v2"}}