{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/7265","labels_url":"https://api.github.com/repos/apache/spark/issues/7265/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/7265/comments","events_url":"https://api.github.com/repos/apache/spark/issues/7265/events","html_url":"https://github.com/apache/spark/pull/7265","id":93622548,"number":7265,"title":"[SPARK-7263] Add new shuffle manager which stores shuffle blocks in Parquet","user":{"login":"massie","id":19554,"avatar_url":"https://avatars.githubusercontent.com/u/19554?v=3","gravatar_id":"","url":"https://api.github.com/users/massie","html_url":"https://github.com/massie","followers_url":"https://api.github.com/users/massie/followers","following_url":"https://api.github.com/users/massie/following{/other_user}","gists_url":"https://api.github.com/users/massie/gists{/gist_id}","starred_url":"https://api.github.com/users/massie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/massie/subscriptions","organizations_url":"https://api.github.com/users/massie/orgs","repos_url":"https://api.github.com/users/massie/repos","events_url":"https://api.github.com/users/massie/events{/privacy}","received_events_url":"https://api.github.com/users/massie/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":48,"created_at":"2015-07-07T20:37:18Z","updated_at":"2015-07-14T21:48:31Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/7265","html_url":"https://github.com/apache/spark/pull/7265","diff_url":"https://github.com/apache/spark/pull/7265.diff","patch_url":"https://github.com/apache/spark/pull/7265.patch"},"body":"This commit adds a new Spark shuffle manager which reads and writes shuffle data to Apache Parquet files. Parquet has a File interface (not a streaming interface) because it is column-oriented and seeks in a File for metadata information, e.g. schemas, statistics. As such, this implementation fetches remote data to local, temporary blocks before the data is passed to Parquet for reading.\r\n    \r\nThis managers uses the following spark configuration parameters to configure Parquet: `spark.shuffle.parquet.{compression, blocksize, pagesize, enabledictionary}.`\r\n\r\nThere is a `spark.shuffle.parquet.fallback` configuration option which allows users to specify a fallback shuffle manager. If the Parquet manager finds that the classes being shuffled have no schema information, and therefore can't be used, it will fallback to the specified fallback manager. With this PR, only Avro `IndexedRecords` are supported in the Parquet shuffle; however, it is straight-forward to extend this to other serialization systems that Parquet supports, e.g. Apache Thrift. If there is no `spark.shuffle.parquet.fallback` defined, any shuffle objects which are not compatible with Parquet will cause an error to be thrown which lists the incompatible objects.\r\n\r\nBecause the `ShuffleDependency` forwards the key, value and combined class information, a full schema can be generated before the first read/write. This allows for less errors (since reflection isn't used) and makes support for null values possible without complex code.\r\n\r\nThe `ExternalSorter`, if needed, is setup to not spill to disk if Parquet is used. In the future, an `ExternalSorter` would need to be created that can read/write Parquet.\r\n\r\nOnly record-level metrics are supported at this time. Byte-level metrics are not currently supported and are complicated somewhat by column compression."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/121401865","html_url":"https://github.com/apache/spark/pull/7265#issuecomment-121401865","issue_url":"https://api.github.com/repos/apache/spark/issues/7265","id":121401865,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-07-14T21:48:31Z","updated_at":"2015-07-14T21:48:31Z","body":"  [Test build #37270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37270/consoleFull) for   PR 7265 at commit [`56be3a2`](https://github.com/apache/spark/commit/56be3a22a8d1422450fe943252ab4d73aebbab5c)."}}