{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/3858","labels_url":"https://api.github.com/repos/apache/spark/issues/3858/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/3858/comments","events_url":"https://api.github.com/repos/apache/spark/issues/3858/events","html_url":"https://github.com/apache/spark/pull/3858","id":53194528,"number":3858,"title":"[SPARK-5037][STREAMING] dynamically loaded DStreams implementation and example","user":{"login":"industrial-sloth","id":8592708,"avatar_url":"https://avatars.githubusercontent.com/u/8592708?v=3","gravatar_id":"","url":"https://api.github.com/users/industrial-sloth","html_url":"https://github.com/industrial-sloth","followers_url":"https://api.github.com/users/industrial-sloth/followers","following_url":"https://api.github.com/users/industrial-sloth/following{/other_user}","gists_url":"https://api.github.com/users/industrial-sloth/gists{/gist_id}","starred_url":"https://api.github.com/users/industrial-sloth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/industrial-sloth/subscriptions","organizations_url":"https://api.github.com/users/industrial-sloth/orgs","repos_url":"https://api.github.com/users/industrial-sloth/repos","events_url":"https://api.github.com/users/industrial-sloth/events{/privacy}","received_events_url":"https://api.github.com/users/industrial-sloth/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2014-12-31T17:38:35Z","updated_at":"2015-07-10T01:03:02Z","closed_at":"2015-07-10T00:15:27Z","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/3858","html_url":"https://github.com/apache/spark/pull/3858","diff_url":"https://github.com/apache/spark/pull/3858.diff","patch_url":"https://github.com/apache/spark/pull/3858.patch"},"body":"This PR adds a new reflection-based method of creating input DStreams to the scala StreamingContext, and wires it through to the python streaming API.\r\n\r\nTrying to create DStream instances directly by reflection runs into trouble with unwanted stuff getting dragged into closures, so I worked around this by defining a new abstract serializable `ReflectedDStreamFactory` class. The idea is that one subclasses this with a concrete implementation that directly instantiates the desired InputDStream; then the StreamingContext uses reflection to dynamically load this new Factory implementation. This PR also has an example showing how this works with the existing ZeroMQ example code in both the scala and python streaming APIs.\r\n\r\nParameters are passed into the input DStream indirectly by first putting them into the factory constructor, then requiring the factory implementation to pass them on into the DStream instance. At the moment these parameters are limited to String type, which I think should cover the majority of use cases, but I'd think it should be possible to generalize this further. \r\n\r\nAm throwing this out there for comment; suggestions and alternative approaches more than welcome."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/120189769","html_url":"https://github.com/apache/spark/pull/3858#issuecomment-120189769","issue_url":"https://api.github.com/repos/apache/spark/issues/3858","id":120189769,"user":{"login":"tdas","id":663212,"avatar_url":"https://avatars.githubusercontent.com/u/663212?v=3","gravatar_id":"","url":"https://api.github.com/users/tdas","html_url":"https://github.com/tdas","followers_url":"https://api.github.com/users/tdas/followers","following_url":"https://api.github.com/users/tdas/following{/other_user}","gists_url":"https://api.github.com/users/tdas/gists{/gist_id}","starred_url":"https://api.github.com/users/tdas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdas/subscriptions","organizations_url":"https://api.github.com/users/tdas/orgs","repos_url":"https://api.github.com/users/tdas/repos","events_url":"https://api.github.com/users/tdas/events{/privacy}","received_events_url":"https://api.github.com/users/tdas/received_events","type":"User","site_admin":false},"created_at":"2015-07-10T01:03:02Z","updated_at":"2015-07-10T01:03:02Z","body":"Okay. I hope you get time in future.\n\nTD\n\nOn Thu, Jul 9, 2015 at 5:16 PM, industrial-sloth <notifications@github.com>\nwrote:\n\n> Closed #3858 <https://github.com/apache/spark/pull/3858>.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/apache/spark/pull/3858#event-352450416>.\n>\n"}}