{"push_id":815875147,"size":29,"distinct_size":29,"ref":"refs/heads/master","head":"883bd8fccf83aae7a2a847c9a6ca129fac86e6a3","before":"9b9fe5f7bf55257269d8febcd64e95677075dfb6","commits":[{"sha":"dba95ea03216e6b8e623db4a36e1018c6ed95538","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-10825] [CORE] [TESTS] Fix race conditions in StandaloneDynamicAllocationSuite\n\nFix the following issues in StandaloneDynamicAllocationSuite:\n\n1. It should not assume master and workers start in order\n2. It should not assume master and workers get ready at once\n3. It should not assume the application is already registered with master after creating SparkContext\n4. It should not access Master.app and idToApp which are not thread safe\n\nThe changes includes:\n* Use `eventually` to wait until master and workers are ready to fix 1 and 2\n* Use `eventually`  to wait until the application is registered with master to fix 3\n* Use `askWithRetry[MasterStateResponse](RequestMasterState)` to get the application info to fix 4\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #8914 from zsxwing/fix-StandaloneDynamicAllocationSuite.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/dba95ea03216e6b8e623db4a36e1018c6ed95538"},{"sha":"b7ad54ec793af1c84973b402f5cceb88307f7996","author":{"email":"45d0cd84ae16b71953c1beb3be9ef57773a10b25@gmail.com","name":"Ryan Williams"},"message":"[SPARK-10871] include number of executor failures in error msg\n\nAuthor: Ryan Williams <ryan.blake.williams@gmail.com>\n\nCloses #8939 from ryan-williams/errmsg.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/b7ad54ec793af1c84973b402f5cceb88307f7996"},{"sha":"ab41864f91713b450695babd5c1424622cb57a54","author":{"email":"1d6e1cf70ec6f9ab28d3ea4b27a49a77654d370e@noelsmith.com","name":"noelsmith"},"message":"[SPARK-10415] [PYSPARK] [MLLIB] [DOCS] Enhance Navigation Sidebar in PySpark API\n\nThese are CSS/JavaScript changes changes to make navigation in the PySpark API a bit simpler by adding the following to the sidebar:\n\n* Classes\n* Functions\n* Tags to highlight experimental features\n\n![screen shot 2015-09-02 at 08 50 12](https://cloud.githubusercontent.com/assets/11915197/9634781/301f853a-518b-11e5-8d5c-fda202f6202f.png)\n\nOnline example here: https://dl.dropboxusercontent.com/u/20821334/pyspark-api-nav-enhance/pyspark.mllib.html\n\n(The contribution is my original work and that I license the work to the project under the project's open source license)\n\nAuthor: noelsmith <mail@noelsmith.com>\n\nCloses #8571 from noel-smith/pyspark-api-nav-enhance.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/ab41864f91713b450695babd5c1424622cb57a54"},{"sha":"7d399c9daa6769ab234890c551e1b3456e0e6e85","author":{"email":"682ed9c9c9639195d4aede92e11bf20fb7db3435@opower.com","name":"Erik Shilts"},"message":"[SPARK-6919] [PYSPARK] Add asDict method to StatCounter\n\nAdd method to easily convert a StatCounter instance into a Python dict\n\nhttps://issues.apache.org/jira/browse/SPARK-6919\n\nNote: This is my original work and the existing Spark license applies.\n\nAuthor: Erik Shilts <erik.shilts@opower.com>\n\nCloses #5516 from eshilts/statcounter-asdict.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/7d399c9daa6769ab234890c551e1b3456e0e6e85"},{"sha":"c1ad373f26053e1906fce7681c03d130a642bf33","author":{"email":"c9068c1a8d85a25509fa3c37c1be36e867289542@intel.com","name":"asokadiggs"},"message":"[SPARK-10782] [PYTHON] Update dropDuplicates documentation\n\nDocumentation for dropDuplicates() and drop_duplicates() is one and the same.  Resolved the error in the example for drop_duplicates using the same approach used for groupby and groupBy, by indicating that dropDuplicates and drop_duplicates are aliases.\n\nAuthor: asokadiggs <asoka.diggs@intel.com>\n\nCloses #8930 from asokadiggs/jira-10782.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/c1ad373f26053e1906fce7681c03d130a642bf33"},{"sha":"4d5a005b0d2591d4d57a19be48c4954b9f1434a9","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-10811] [SQL] Eliminates unnecessary byte array copying\n\nWhen reading Parquet string and binary-backed decimal values, Parquet `Binary.getBytes` always returns a copied byte array, which is unnecessary. Since the underlying implementation of `Binary` values there is guaranteed to be `ByteArraySliceBackedBinary`, and Parquet itself never reuses underlying byte arrays, we can use `Binary.toByteBuffer.array()` to steal the underlying byte arrays without copying them.\n\nThis brings performance benefits when scanning Parquet string and binary-backed decimal columns. Note that, this trick doesn't cover binary-backed decimals with precision greater than 18.\n\nMy micro-benchmark result is that, this brings a ~15% performance boost for scanning TPC-DS `store_sales` table (scale factor 15).\n\nAnother minor optimization done in this PR is that, now we directly construct a Java `BigDecimal` in `Decimal.toJavaBigDecimal` without constructing a Scala `BigDecimal` first. This brings another ~5% performance gain.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #8907 from liancheng/spark-10811/eliminate-array-copying.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/4d5a005b0d2591d4d57a19be48c4954b9f1434a9"},{"sha":"2931e89f0c54248d87f1f84c81137a5a91e142e9","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-10736] [ML] Use 1 for all ratings if $(ratingCol) = \"\"\n\nFor some implicit dataset, ratings may not exist in the training data. In this case, we can assume all observed pairs to be positive and treat their ratings as 1. This should happen when users set ```ratingCol``` to an empty string.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8937 from yanboliang/spark-10736.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/2931e89f0c54248d87f1f84c81137a5a91e142e9"},{"sha":"16fd2a2f4225771af43403f20fbc79b6c914ee4d","author":{"email":"679c4ded150c6b9a8052807f108ce836961fed8d@questtec.nl","name":"Herman van Hovell"},"message":"[SPARK-9741] [SQL] Approximate Count Distinct using the new UDAF interface.\n\nThis PR implements a HyperLogLog based Approximate Count Distinct function using the new UDAF interface.\n\nThe implementation is inspired by the ClearSpring HyperLogLog implementation and should produce the same results.\n\nThere is still some documentation and testing left to do.\n\ncc yhuai\n\nAuthor: Herman van Hovell <hvanhovell@questtec.nl>\n\nCloses #8362 from hvanhovell/SPARK-9741.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/16fd2a2f4225771af43403f20fbc79b6c914ee4d"},{"sha":"c7b29ae6418368a1266b960ba8776317fd867313","author":{"email":"3f27eab2afd49386f0ac811afb0bc245aa92c0fa@intel.com","name":"Sun Rui"},"message":"[SPARK-10851] [SPARKR] Exception not failing R applications (in yarn cluster mode)\n\nThe YARN backend doesn't like when user code calls System.exit, since it cannot know the exit status and thus cannot set an appropriate final status for the application.\n\nThis PR remove the usage of system.exit to exit the RRunner. Instead, when the R process running an SparkR script returns an exit code other than 0, throws SparkUserAppException which will be caught by ApplicationMaster and ApplicationMaster knows it failed. For other failures, throws SparkException.\n\nAuthor: Sun Rui <rui.sun@intel.com>\n\nCloses #8938 from sun-rui/SPARK-10851.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/c7b29ae6418368a1266b960ba8776317fd867313"},{"sha":"03cca5dce2cd7618b5c0e33163efb8502415b06e","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-10770] [SQL] SparkPlan.executeCollect/executeTake should return InternalRow rather than external Row.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #8900 from rxin/SPARK-10770-1.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/03cca5dce2cd7618b5c0e33163efb8502415b06e"},{"sha":"89ea0041ae5a701ce8d211ed08f1f059b7f9c396","author":{"email":"675f9d3e0a4a073d3f727b49da33adefb2281ad7@godaddy.com","name":"Nathan Howell"},"message":"[SPARK-9617] [SQL] Implement json_tuple\n\nThis is an implementation of Hive's `json_tuple` function using Jackson Streaming.\n\nAuthor: Nathan Howell <nhowell@godaddy.com>\n\nCloses #7946 from NathanHowell/SPARK-9617.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/89ea0041ae5a701ce8d211ed08f1f059b7f9c396"},{"sha":"f21e2da03fbf8041fece476e3d5c699aef819451","author":{"email":"1f62a455e81276e762523a0dbb882128bcd8957f@mail.usf.edu","name":"Oscar D. Lara Yejas"},"message":"[SPARK-10807] [SPARKR] Added as.data.frame as a synonym for collect\n\nCreated method as.data.frame as a synonym for collect().\n\nAuthor: Oscar D. Lara Yejas <olarayej@mail.usf.edu>\nAuthor: olarayej <oscar.lara.yejas@us.ibm.com>\nAuthor: Oscar D. Lara Yejas <oscar.lara.yejas@us.ibm.com>\n\nCloses #8908 from olarayej/SPARK-10807.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/f21e2da03fbf8041fece476e3d5c699aef819451"},{"sha":"9b3e7768a27d51ddd4711c4a68a428a6875bd6d7","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-10058] [CORE] [TESTS] Fix the flaky tests in HeartbeatReceiverSuite\n\nFixed the test failure here: https://amplab.cs.berkeley.edu/jenkins/view/Spark-QA-Test/job/Spark-1.5-SBT/116/AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.2,label=spark-test/testReport/junit/org.apache.spark/HeartbeatReceiverSuite/normal_heartbeat/\n\nThis failure is because `HeartbeatReceiverSuite. heartbeatReceiver` may receive `SparkListenerExecutorAdded(\"driver\")` sent from [LocalBackend](https://github.com/apache/spark/blob/8fb3a65cbb714120d612e58ef9d12b0521a83260/core/src/main/scala/org/apache/spark/scheduler/local/LocalBackend.scala#L121).\n\nThere are other race conditions in `HeartbeatReceiverSuite` because `HeartbeatReceiver.onExecutorAdded` and `HeartbeatReceiver.onExecutorRemoved` are asynchronous. This PR also fixed them.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #8946 from zsxwing/SPARK-10058.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/9b3e7768a27d51ddd4711c4a68a428a6875bd6d7"},{"sha":"4d8c7c6d1c973f07e210e27936b063b5a763e9a3","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-10865] [SPARK-10866] [SQL] Fix bug of ceil/floor, which should returns long instead of the Double type\n\nFloor & Ceiling function should returns Long type, rather than Double.\n\nVerified with MySQL & Hive.\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #8933 from chenghao-intel/ceiling.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/4d8c7c6d1c973f07e210e27936b063b5a763e9a3"},{"sha":"02026a8132a2e6fe3be97b33b49826139cd1312e","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@163.com","name":"Wenchen Fan"},"message":"[SPARK-10671] [SQL] Throws an analysis exception if we cannot find Hive UDFs\n\nTakes over https://github.com/apache/spark/pull/8800\n\nAuthor: Wenchen Fan <cloud0fan@163.com>\n\nCloses #8941 from cloud-fan/hive-udf.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/02026a8132a2e6fe3be97b33b49826139cd1312e"},{"sha":"01cd688f5245cbb752863100b399b525b31c3510","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-10400] [SQL] Renames SQLConf.PARQUET_FOLLOW_PARQUET_FORMAT_SPEC\n\nWe introduced SQL option `spark.sql.parquet.followParquetFormatSpec` while working on implementing Parquet backwards-compatibility rules in SPARK-6777. It indicates whether we should use legacy Parquet format adopted by Spark 1.4 and prior versions or the standard format defined in parquet-format spec to write Parquet files.\n\nThis option defaults to `false` and is marked as a non-public option (`isPublic = false`) because we haven't finished refactored Parquet write path. The problem is, the name of this option is somewhat confusing, because it's not super intuitive why we shouldn't follow the spec. Would be nice to rename it to `spark.sql.parquet.writeLegacyFormat`, and invert its default value (the two option names have opposite meanings).\n\nAlthough this option is private in 1.5, we'll make it public in 1.6 after refactoring Parquet write path. So that users can decide whether to write Parquet files in standard format or legacy format.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #8566 from liancheng/spark-10400/deprecate-follow-parquet-format-spec.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/01cd688f5245cbb752863100b399b525b31c3510"},{"sha":"2272962eb087ffedaee12c761506e33e45bd0239","author":{"email":"e58867706a26c17cde71f0f9f5e010ffb994f9e8@gmail.com","name":"Takeshi YAMAMURO"},"message":"[SPARK-9867] [SQL] Move utilities for binary data into ByteArray\n\nThe utilities such as Substring#substringBinarySQL and BinaryPrefixComparator#computePrefix for binary data are put together in ByteArray for easy-to-read.\n\nAuthor: Takeshi YAMAMURO <linguin.m.s@gmail.com>\n\nCloses #8122 from maropu/CleanUpForBinaryType.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/2272962eb087ffedaee12c761506e33e45bd0239"},{"sha":"2a717821bbb026d4d8c43d31b3300721357951c6","author":{"email":"5ae1e08e9cc729c7cf8bab04d7dd3d8c9b1e9ba9@kth.se","name":"Rerngvit Yanggratoke"},"message":"[SPARK-9798] [ML] CrossValidatorModel Documentation Improvements\n\nDocument CrossValidatorModel members: bestModel and avgMetrics\n\nAuthor: Rerngvit Yanggratoke <rerngvit@kth.se>\n\nCloses #8882 from rerngvit/Spark-9798.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/2a717821bbb026d4d8c43d31b3300721357951c6"},{"sha":"23a9448c04da7130d6c41c37f9fdf03184422dc8","author":{"email":"7853928d03425cf3be35be2fa0c5b99641435d26@gmail.com","name":"Xusen Yin"},"message":"[SPARK-5890] [ML] Add feature discretizer\n\nJIRA issue [here](https://issues.apache.org/jira/browse/SPARK-5890).\n\nI borrow the code of `findSplits` from `RandomForest`. I don't think it's good to call it from `RandomForest` directly.\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\n\nCloses #5779 from yinxusen/SPARK-5890.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/23a9448c04da7130d6c41c37f9fdf03184422dc8"},{"sha":"633aaae0a1e31e9ba634423840e350b22342c6b5","author":{"email":"7853928d03425cf3be35be2fa0c5b99641435d26@gmail.com","name":"Xusen Yin"},"message":"[SPARK-6530] [ML] Add chi-square selector for ml package\n\nSee JIRA [here](https://issues.apache.org/jira/browse/SPARK-6530).\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\n\nCloses #5742 from yinxusen/SPARK-6530.","distinct":true,"url":"https://api.github.com/repos/jasoncl/spark/commits/633aaae0a1e31e9ba634423840e350b22342c6b5"}]}