{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8635","labels_url":"https://api.github.com/repos/apache/spark/issues/8635/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8635/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8635/events","html_url":"https://github.com/apache/spark/pull/8635","id":105126748,"number":8635,"title":"[SPARK-10466][SQL] UnsafeRow SerDe exception with data spill","user":{"login":"chenghao-intel","id":2367031,"avatar_url":"https://avatars.githubusercontent.com/u/2367031?v=3","gravatar_id":"","url":"https://api.github.com/users/chenghao-intel","html_url":"https://github.com/chenghao-intel","followers_url":"https://api.github.com/users/chenghao-intel/followers","following_url":"https://api.github.com/users/chenghao-intel/following{/other_user}","gists_url":"https://api.github.com/users/chenghao-intel/gists{/gist_id}","starred_url":"https://api.github.com/users/chenghao-intel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chenghao-intel/subscriptions","organizations_url":"https://api.github.com/users/chenghao-intel/orgs","repos_url":"https://api.github.com/users/chenghao-intel/repos","events_url":"https://api.github.com/users/chenghao-intel/events{/privacy}","received_events_url":"https://api.github.com/users/chenghao-intel/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-09-07T00:49:52Z","updated_at":"2015-09-09T04:32:09Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8635","html_url":"https://github.com/apache/spark/pull/8635","diff_url":"https://github.com/apache/spark/pull/8635.diff","patch_url":"https://github.com/apache/spark/pull/8635.patch"},"body":"Data Spill with UnsafeRow causes assert failure.\r\n\r\n```\r\njava.lang.AssertionError: assertion failed\r\n\tat scala.Predef$.assert(Predef.scala:165)\r\n\tat org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$2.writeKey(UnsafeRowSerializer.scala:75)\r\n\tat org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:180)\r\n\tat org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$2$$anonfun$apply$1.apply(ExternalSorter.scala:688)\r\n\tat org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$2$$anonfun$apply$1.apply(ExternalSorter.scala:687)\r\n\tat scala.collection.Iterator$class.foreach(Iterator.scala:727)\r\n\tat scala.collection.AbstractIterator.foreach(Iterator.scala:1157)\r\n\tat org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$2.apply(ExternalSorter.scala:687)\r\n\tat org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$2.apply(ExternalSorter.scala:683)\r\n\tat scala.collection.Iterator$class.foreach(Iterator.scala:727)\r\n\tat scala.collection.AbstractIterator.foreach(Iterator.scala:1157)\r\n\tat org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:683)\r\n\tat org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:80)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:88)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\r\n```\r\n\r\nTo reproduce that with code (thanks @andrewor14):\r\n```scala\r\nbin/spark-shell --master local\r\n  --conf spark.shuffle.memoryFraction=0.005\r\n  --conf spark.shuffle.sort.bypassMergeThreshold=0\r\n\r\nsc.parallelize(1 to 2 * 1000 * 1000, 10)\r\n  .map { i => (i, i) }.toDF(\"a\", \"b\").groupBy(\"b\").avg().count()\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/138777055","html_url":"https://github.com/apache/spark/pull/8635#issuecomment-138777055","issue_url":"https://api.github.com/repos/apache/spark/issues/8635","id":138777055,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-09-09T04:32:09Z","updated_at":"2015-09-09T04:32:09Z","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42178/\nTest PASSed."}}