{"action":"created","issue":{"url":"https://api.github.com/repos/amplab/keystone/issues/151","labels_url":"https://api.github.com/repos/amplab/keystone/issues/151/labels{/name}","comments_url":"https://api.github.com/repos/amplab/keystone/issues/151/comments","events_url":"https://api.github.com/repos/amplab/keystone/issues/151/events","html_url":"https://github.com/amplab/keystone/issues/151","id":93222963,"number":151,"title":"Should I switch from spark.ml.pipeline to kestone for computer vision task ?","user":{"login":"jrabary","id":1025387,"avatar_url":"https://avatars.githubusercontent.com/u/1025387?v=3","gravatar_id":"","url":"https://api.github.com/users/jrabary","html_url":"https://github.com/jrabary","followers_url":"https://api.github.com/users/jrabary/followers","following_url":"https://api.github.com/users/jrabary/following{/other_user}","gists_url":"https://api.github.com/users/jrabary/gists{/gist_id}","starred_url":"https://api.github.com/users/jrabary/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jrabary/subscriptions","organizations_url":"https://api.github.com/users/jrabary/orgs","repos_url":"https://api.github.com/users/jrabary/repos","events_url":"https://api.github.com/users/jrabary/events{/privacy}","received_events_url":"https://api.github.com/users/jrabary/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-07-06T08:30:02Z","updated_at":"2015-07-06T15:28:32Z","closed_at":"2015-07-06T15:28:32Z","body":"Dear all,\r\n\r\nAs working in the field of large scale computer vision, I find keystoneML very interesting. I began to implement myself a computer vision task pipeline based on the ampcamp 5 pipeline demo but using the API of spark.ml. I have now some computer vision feature extractor that are subclass of spark.ml.Transformer and use spark.ml.Pipeline to solve my problems. But, in keystone, there are much more interesting feature than I have and I'm thinking to switch to keystone in order to don't reinvent  the wheel. So my question is : are these two project will be merged one day and what the main difference between keystone and spark.ml ? \r\nWith spark.ml.Pipeline one works directly with DataFrame. With keystone, the pipeline transforms RDDs. Is it more efficient ? \r\n\r\nJao"},"comment":{"url":"https://api.github.com/repos/amplab/keystone/issues/comments/118891193","html_url":"https://github.com/amplab/keystone/issues/151#issuecomment-118891193","issue_url":"https://api.github.com/repos/amplab/keystone/issues/151","id":118891193,"user":{"login":"etrain","id":1326181,"avatar_url":"https://avatars.githubusercontent.com/u/1326181?v=3","gravatar_id":"","url":"https://api.github.com/users/etrain","html_url":"https://github.com/etrain","followers_url":"https://api.github.com/users/etrain/followers","following_url":"https://api.github.com/users/etrain/following{/other_user}","gists_url":"https://api.github.com/users/etrain/gists{/gist_id}","starred_url":"https://api.github.com/users/etrain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/etrain/subscriptions","organizations_url":"https://api.github.com/users/etrain/orgs","repos_url":"https://api.github.com/users/etrain/repos","events_url":"https://api.github.com/users/etrain/events{/privacy}","received_events_url":"https://api.github.com/users/etrain/received_events","type":"User","site_admin":false},"created_at":"2015-07-06T15:28:32Z","updated_at":"2015-07-06T15:28:32Z","body":"Thanks for stopping by @jrabary. This is probably a discussion better had on our users mailing list (keystoneml-users@groups.google.com). \r\n\r\nIn fact, a question like this one already came up and was addressed there: https://groups.google.com/forum/#!topic/keystoneml-users/HHaYgkJlDSM\r\n\r\nThere are no concrete plans to merge the two projects right now - KeystoneML is a research project, but we're very interested in making sure we can support common workloads. Ideas and/or code from KeystoneML may make their way into spark.ml over time.\r\n\r\nIn terms of efficiency - it is tough to say whether DataFrame is more or less efficient than native RDD operations. This will depend on a number of factors, including the native data representation under the covers and whether a pipeline can be optimized by the SparkSQL optimizer. We feel that the RDD-oriented transformations provide the ability to make bulk operations more efficient, and the type-safety provides compile-time assurances about whether a pipeline will actually run or not. That said - this is pretty hand-wavy and it would make more sense to talk about a concrete example if you have one."}}