{"push_id":798156568,"size":13,"distinct_size":13,"ref":"refs/heads/master","head":"d83b6aae8b4357c56779cc98804eb350ab8af62d","before":"74d8f7dda82c3a16348f3ff22da83203e0b7f708","commits":[{"sha":"e3b5d6cb29e0f983fcc55920619e6433298955f5","author":{"email":"04ade2e3a4ce65c5ee0cbfa08dacc69e7d690176@apache.org","name":"navis.ryu"},"message":"[SPARK-10684] [SQL] StructType.interpretedOrdering need not to be serialized\n\nKryo fails with buffer overflow even with max value (2G).\n\n{noformat}\norg.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 1\nSerialization trace:\ncontainsChild (org.apache.spark.sql.catalyst.expressions.BoundReference)\nchild (org.apache.spark.sql.catalyst.expressions.SortOrder)\narray (scala.collection.mutable.ArraySeq)\nordering (org.apache.spark.sql.catalyst.expressions.InterpretedOrdering)\ninterpretedOrdering (org.apache.spark.sql.types.StructType)\nschema (org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema). To avoid this, increase spark.kryoserializer.buffer.max value.\n        at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:263)\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:240)\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n        at java.lang.Thread.run(Thread.java:745)\n{noformat}\n\nAuthor: navis.ryu <navis@apache.org>\n\nCloses #8808 from navis/SPARK-10684.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/e3b5d6cb29e0f983fcc55920619e6433298955f5"},{"sha":"20fd35dfd1ac402b622604e7bbedcc53a580b0a2","author":{"email":"808d31295a27ded55339ed818ba0b1912ca15edb@guavus.com","name":"Yash Datta"},"message":"[SPARK-10451] [SQL] Prevent unnecessary serializations in InMemoryColumnarTableScan\n\nMany of the fields in InMemoryColumnar scan and InMemoryRelation can be made transient.\n\nThis  reduces my 1000ms job to abt 700 ms . The task size reduces from 2.8 mb to ~1300kb\n\nAuthor: Yash Datta <Yash.Datta@guavus.com>\n\nCloses #8604 from saucam/serde.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/20fd35dfd1ac402b622604e7bbedcc53a580b0a2"},{"sha":"35e8ab939000d4a1a01c1af4015c25ff6f4013a3","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-10615] [PYSPARK] change assertEquals to assertEqual\n\nAs ```assertEquals``` is deprecated, so we need to change ```assertEquals``` to ```assertEqual``` for existing python unit tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8814 from yanboliang/spark-10615.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/35e8ab939000d4a1a01c1af4015c25ff6f4013a3"},{"sha":"00a2911c5bea67a1a4796fb1d6fd5d0a8ee79001","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-10540] Fixes flaky all-data-type test\n\nThis PR breaks the original test case into multiple ones (one test case for each data type). In this way, test failure output can be much more readable.\n\nWithin each test case, we build a table with two columns, one of them is for the data type to test, the other is an \"index\" column, which is used to sort the DataFrame and workaround [SPARK-10591] [1]\n\n[1]: https://issues.apache.org/jira/browse/SPARK-10591\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #8768 from liancheng/spark-10540/test-all-data-types.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/00a2911c5bea67a1a4796fb1d6fd5d0a8ee79001"},{"sha":"c6f8135ee52202bd86adb090ab631e80330ea4df","author":{"email":"348ff379544b45aa3f16e53a543620141423a0f6@gmail.com","name":"Yijie Shen"},"message":"[SPARK-10539] [SQL] Project should not be pushed down through Intersect or Except #8742\n\nIntersect and Except are both set operators and they use the all the columns to compare equality between rows. When pushing their Project parent down, the relations they based on would change, therefore not an equivalent transformation.\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-10539\n\nI added some comments based on the fix of https://github.com/apache/spark/pull/8742.\n\nAuthor: Yijie Shen <henry.yijieshen@gmail.com>\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #8823 from yhuai/fix_set_optimization.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/c6f8135ee52202bd86adb090ab631e80330ea4df"},{"sha":"3a22b1004f527d54d399dd0225cd7f2f8ffad9c5","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-10449] [SQL] Don't merge decimal types with incompatable precision or scales\n\nFrom JIRA: Schema merging should only handle struct fields. But currently we also reconcile decimal precision and scale information.\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #8634 from holdenk/SPARK-10449-dont-merge-different-precision.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/3a22b1004f527d54d399dd0225cd7f2f8ffad9c5"},{"sha":"348d7c9a93dd00d3d1859342a8eb0aea2e77f597","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9808] Remove hash shuffle file consolidation.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #8812 from rxin/SPARK-9808-1.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/348d7c9a93dd00d3d1859342a8eb0aea2e77f597"},{"sha":"8074208fa47fa654c1055c48cfa0d923edeeb04f","author":{"email":"47d77dc6a54bed715112a2a0201e3ffd984fed82@palantir.com","name":"Mingyu Kim"},"message":"[SPARK-10611] Clone Configuration for each task for NewHadoopRDD\n\nThis patch attempts to fix the Hadoop Configuration thread safety issue for NewHadoopRDD in the same way SPARK-2546 fixed the issue for HadoopRDD.\n\nAuthor: Mingyu Kim <mkim@palantir.com>\n\nCloses #8763 from mingyukim/mkim/SPARK-10611.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/8074208fa47fa654c1055c48cfa0d923edeeb04f"},{"sha":"c8149ef2c57f5c47ab97ee8d8d58a216d4bc4156","author":{"email":"cc463b26613f4ba480f0f94ca5cb2cab678c4fcb@databricks.com","name":"Eric Liang"},"message":"[MINOR] [ML] override toString of AttributeGroup\n\nThis makes equality test failures much more readable.\n\nmengxr\n\nAuthor: Eric Liang <ekl@databricks.com>\nAuthor: Eric Liang <ekhliang@gmail.com>\n\nCloses #8826 from ericl/attrgroupstr.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/c8149ef2c57f5c47ab97ee8d8d58a216d4bc4156"},{"sha":"22be2ae147a111e88896f6fb42ed46bbf108a99b","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-10623] [SQL] Fixes ORC predicate push-down\n\nWhen pushing down a leaf predicate, ORC `SearchArgument` builder requires an extra \"parent\" predicate (any one among `AND`/`OR`/`NOT`) to wrap the leaf predicate. E.g., to push down `a < 1`, we must build `AND(a < 1)` instead. Fortunately, when actually constructing the `SearchArgument`, the builder will eliminate all those unnecessary wrappers.\n\nThis PR is based on #8783 authored by zhzhan. I also took the chance to simply `OrcFilters` a little bit to improve readability.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #8799 from liancheng/spark-10623/fix-orc-ppd.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/22be2ae147a111e88896f6fb42ed46bbf108a99b"},{"sha":"7ff8d68cc19299e16dedfd819b9e96480fa6cf44","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-10474] [SQL] Aggregation fails to allocate memory for pointer array\n\nWhen `TungstenAggregation` hits memory pressure, it switches from hash-based to sort-based aggregation in-place. However, in the process we try to allocate the pointer array for writing to the new `UnsafeExternalSorter` *before* actually freeing the memory from the hash map. This lead to the following exception:\n```\n java.io.IOException: Could not acquire 65536 bytes of memory\n        at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.initializeForWriting(UnsafeExternalSorter.java:169)\n        at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.spill(UnsafeExternalSorter.java:220)\n        at org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:126)\n        at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.destructAndCreateExternalSorter(UnsafeFixedWidthAggregationMap.java:257)\n        at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.switchToSortBasedAggregation(TungstenAggregationIterator.scala:435)\n```\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #8827 from andrewor14/allocate-pointer-array.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/7ff8d68cc19299e16dedfd819b9e96480fa6cf44"},{"sha":"d507f9c0b7f7a524137a694ed6443747aaf90463","author":{"email":"7d45abe39fd242ee2f6575da28f13d66568418b3@oss.nttdata.co.jp","name":"Kousuke Saruta"},"message":"[SPARK-10584] [SQL] [DOC] Documentation about the compatible Hive version is wrong.\n\nIn Spark 1.5.0, Spark SQL is compatible with Hive 0.12.0 through 1.2.1 but the documentation is wrong.\n\n/CC yhuai\n\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #8776 from sarutak/SPARK-10584-2.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/d507f9c0b7f7a524137a694ed6443747aaf90463"},{"sha":"d83b6aae8b4357c56779cc98804eb350ab8af62d","author":{"email":"b32196ea7ef6ab9a716eebbe6dcb40fc80478f8a@gmail.com","name":"Alexis Seigneurin"},"message":"Fixed links to the API\n\nSubmitting this change on the master branch as requested in https://github.com/apache/spark/pull/8819#issuecomment-141505941\n\nAuthor: Alexis Seigneurin <alexis.seigneurin@gmail.com>\n\nCloses #8838 from aseigneurin/patch-2.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/d83b6aae8b4357c56779cc98804eb350ab8af62d"}]}