{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/9206","labels_url":"https://api.github.com/repos/apache/spark/issues/9206/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/9206/comments","events_url":"https://api.github.com/repos/apache/spark/issues/9206/events","html_url":"https://github.com/apache/spark/pull/9206","id":112701702,"number":9206,"title":"[SPARK-11235] [network] Add ability to stream data using network lib.","user":{"login":"vanzin","id":1694083,"avatar_url":"https://avatars.githubusercontent.com/u/1694083?v=3","gravatar_id":"","url":"https://api.github.com/users/vanzin","html_url":"https://github.com/vanzin","followers_url":"https://api.github.com/users/vanzin/followers","following_url":"https://api.github.com/users/vanzin/following{/other_user}","gists_url":"https://api.github.com/users/vanzin/gists{/gist_id}","starred_url":"https://api.github.com/users/vanzin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanzin/subscriptions","organizations_url":"https://api.github.com/users/vanzin/orgs","repos_url":"https://api.github.com/users/vanzin/repos","events_url":"https://api.github.com/users/vanzin/events{/privacy}","received_events_url":"https://api.github.com/users/vanzin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2015-10-21T23:49:58Z","updated_at":"2015-11-01T00:57:11Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/9206","html_url":"https://github.com/apache/spark/pull/9206","diff_url":"https://github.com/apache/spark/pull/9206.diff","patch_url":"https://github.com/apache/spark/pull/9206.patch"},"body":"The current interface used to fetch shuffle data is not very efficient for\r\nlarge buffers; it requires the receiver to buffer the entirety of the\r\ncontents being downloaded in memory before processing the data.\r\n\r\nTo use the network library to transfer large files (such as those that\r\ncan be added using SparkContext addJar / addFile), this change adds a\r\nmore efficient way of downloding data, by streaming the data and feeding\r\nit to a callback as data arrives.\r\n\r\nThis is achieved by a custom frame decoder that replaces the current netty\r\none; this decoder allows entering a mode where framing is skipped and data\r\nis instead provided directly to a callback. The existing netty classes\r\n(ByteToMessageDecoder and LengthFieldBasedFrameDecoder) could not be reused\r\nsince their semantics do not allow for the interception approach the new\r\ndecoder uses."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/152784108","html_url":"https://github.com/apache/spark/pull/9206#issuecomment-152784108","issue_url":"https://api.github.com/repos/apache/spark/issues/9206","id":152784108,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-11-01T00:57:11Z","updated_at":"2015-11-01T00:57:11Z","body":" Merged build triggered."}}