{"push_id":906087451,"size":72,"distinct_size":72,"ref":"refs/heads/master","head":"6b8e5168ae291a300711f355b1364f5de1135f50","before":"136948303e800120484fde61de331fe35f5a698b","commits":[{"sha":"eeb58722ad73441eeb5f35f864be3c5392cfd426","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-12198][SPARKR] SparkR support read.parquet and deprecate parquetFile\n\nSparkR support ```read.parquet``` and deprecate ```parquetFile```. This change is similar with #10145 for ```jsonFile```.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #10191 from yanboliang/spark-12198.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/eeb58722ad73441eeb5f35f864be3c5392cfd426"},{"sha":"9fba9c8004d2b97549e5456fa7918965bec27336","author":{"email":"66332c93c294b372ab1bdd5acb9b89a538e2d2f2@gmail.com","name":"Yuhao Yang"},"message":"[SPARK-11602][MLLIB] Refine visibility for 1.6 scala API audit\n\njira: https://issues.apache.org/jira/browse/SPARK-11602\n\nMade a pass on the API change of 1.6. Open the PR for efficient discussion.\n\nAuthor: Yuhao Yang <hhbyyh@gmail.com>\n\nCloses #9939 from hhbyyh/auditScala.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/9fba9c8004d2b97549e5456fa7918965bec27336"},{"sha":"d9d354ed40eec56b3f03d32f4e2629d367b1bf02","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-12234][SPARKR] Fix ```subset``` function error when only set ```select``` argument\n\nFix ```subset``` function error when only set ```select``` argument. Please refer to the [JIRA](https://issues.apache.org/jira/browse/SPARK-12234) about the error and how to reproduce it.\n\ncc sun-rui felixcheung shivaram\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #10217 from yanboliang/spark-12234.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/d9d354ed40eec56b3f03d32f4e2629d367b1bf02"},{"sha":"bc5f56aa60a430244ffa0cacd81c0b1ecbf8d68f","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-12250][SQL] Allow users to define a UDAF without providing details of its inputSchema\n\nhttps://issues.apache.org/jira/browse/SPARK-12250\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #10236 from yhuai/SPARK-12250.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/bc5f56aa60a430244ffa0cacd81c0b1ecbf8d68f"},{"sha":"ec5f9ed5de2218938dba52152475daafd4dc4786","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-12228][SQL] Try to run execution hive's derby in memory.\n\nThis PR tries to make execution hive's derby run in memory since it is a fake metastore and every time we create a HiveContext, we will switch to a new one. It is possible that it can reduce the flakyness of our tests that need to create HiveContext (e.g. HiveSparkSubmitSuite). I will test it more.\n\nhttps://issues.apache.org/jira/browse/SPARK-12228\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #10204 from yhuai/derbyInMemory.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/ec5f9ed5de2218938dba52152475daafd4dc4786"},{"sha":"2ecbe02d5b28ee562d10c1735244b90a08532c9e","author":{"email":"5a279bc90c6f3ee7b2e009f7070db93eec368853@databricks.com","name":"Timothy Hunter"},"message":"[SPARK-12212][ML][DOC] Clarifies the difference between spark.ml, spark.mllib and mllib in the documentation.\n\nReplaces a number of occurences of `MLlib` in the documentation that were meant to refer to the `spark.mllib` package instead. It should clarify for new users the difference between `spark.mllib` (the package) and MLlib (the umbrella project for ML in spark).\n\nIt also removes some files that I forgot to delete with #10207\n\nAuthor: Timothy Hunter <timhunter@databricks.com>\n\nCloses #10234 from thunterdb/12212.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/2ecbe02d5b28ee562d10c1735244b90a08532c9e"},{"sha":"550d7a165f4d11bc1741f9ad699311553e0fe195","author":{"email":"1a1ab5abfb65135a4f49eb66e79d9e570e28807d@us.ibm.com","name":"bomeng"},"message":"Merge remote-tracking branch 'upstream/master'","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/550d7a165f4d11bc1741f9ad699311553e0fe195"},{"sha":"4a46b8859d3314b5b45a67cdc5c81fecb6e9e78c","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-11563][CORE][REPL] Use RpcEnv to transfer REPL-generated classes.\n\nThis avoids bringing up yet another HTTP server on the driver, and\ninstead reuses the file server already managed by the driver's\nRpcEnv. As a bonus, the repl now inherits the security features of\nthe network library.\n\nThere's also a small change to create the directory for storing classes\nunder the root temp dir for the application (instead of directly\nunder java.io.tmpdir).\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #9923 from vanzin/SPARK-11563.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/4a46b8859d3314b5b45a67cdc5c81fecb6e9e78c"},{"sha":"6a6c1fc5c807ba4e8aba3e260537aa527ff5d46a","author":{"email":"0510f24c259a4f603c85979def2fe3bfbef1c219@us.ibm.com","name":"Bryan Cutler"},"message":"[SPARK-11713] [PYSPARK] [STREAMING] Initial RDD updateStateByKey for PySpark\n\nAdding ability to define an initial state RDD for use with updateStateByKey PySpark.  Added unit test and changed stateful_network_wordcount example to use initial RDD.\n\nAuthor: Bryan Cutler <bjcutler@us.ibm.com>\n\nCloses #10082 from BryanCutler/initial-rdd-updateStateByKey-SPARK-11713.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/6a6c1fc5c807ba4e8aba3e260537aa527ff5d46a"},{"sha":"23a9e62bad9669e9ff5dc4bd714f58d12f9be0b5","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-12251] Document and improve off-heap memory configurations\n\nThis patch adds documentation for Spark configurations that affect off-heap memory and makes some naming and validation improvements for those configs.\n\n- Change `spark.memory.offHeapSize` to `spark.memory.offHeap.size`. This is fine because this configuration has not shipped in any Spark release yet (it's new in Spark 1.6).\n- Deprecated `spark.unsafe.offHeap` in favor of a new `spark.memory.offHeap.enabled` configuration. The motivation behind this change is to gather all memory-related configurations under the same prefix.\n- Add a check which prevents users from setting `spark.memory.offHeap.enabled=true` when `spark.memory.offHeap.size == 0`. After SPARK-11389 (#9344), which was committed in Spark 1.6, Spark enforces a hard limit on the amount of off-heap memory that it will allocate to tasks. As a result, enabling off-heap execution memory without setting `spark.memory.offHeap.size` will lead to immediate OOMs. The new configuration validation makes this scenario easier to diagnose, helping to avoid user confusion.\n- Document these configurations on the configuration page.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10237 from JoshRosen/SPARK-12251.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/23a9e62bad9669e9ff5dc4bd714f58d12f9be0b5"},{"sha":"5030923ea8bb94ac8fa8e432de9fc7089aa93986","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-12155][SPARK-12253] Fix executor OOM in unified memory management\n\n**Problem.** In unified memory management, acquiring execution memory may lead to eviction of storage memory. However, the space freed from evicting cached blocks is distributed among all active tasks. Thus, an incorrect upper bound on the execution memory per task can cause the acquisition to fail, leading to OOM's and premature spills.\n\n**Example.** Suppose total memory is 1000B, cached blocks occupy 900B, `spark.memory.storageFraction` is 0.4, and there are two active tasks. In this case, the cap on task execution memory is 100B / 2 = 50B. If task A tries to acquire 200B, it will evict 100B of storage but can only acquire 50B because of the incorrect cap. For another example, see this [regression test](https://github.com/andrewor14/spark/blob/fix-oom/core/src/test/scala/org/apache/spark/memory/UnifiedMemoryManagerSuite.scala#L233) that I stole from JoshRosen.\n\n**Solution.** Fix the cap on task execution memory. It should take into account the space that could have been freed by storage in addition to the current amount of memory available to execution. In the example above, the correct cap should have been 600B / 2 = 300B.\n\nThis patch also guards against the race condition (SPARK-12253):\n(1) Existing tasks collectively occupy all execution memory\n(2) New task comes in and blocks while existing tasks spill\n(3) After tasks finish spilling, another task jumps in and puts in a large block, stealing the freed memory\n(4) New task still cannot acquire memory and goes back to sleep\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #10240 from andrewor14/fix-oom.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/5030923ea8bb94ac8fa8e432de9fc7089aa93986"},{"sha":"24d3357d66e14388faf8709b368edca70ea96432","author":{"email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com","name":"jerryshao"},"message":"[STREAMING][DOC][MINOR] Update the description of direct Kafka stream doc\n\nWith the merge of [SPARK-8337](https://issues.apache.org/jira/browse/SPARK-8337), now the Python API has the same functionalities compared to Scala/Java, so here changing the description to make it more precise.\n\nzsxwing tdas , please review, thanks a lot.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #10246 from jerryshao/direct-kafka-doc-update.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/24d3357d66e14388faf8709b368edca70ea96432"},{"sha":"b1b4ee7f3541d92c8bc2b0b4fdadf46cfdb09504","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12258][SQL] passing null into ScalaUDF\n\nCheck nullability and passing them into ScalaUDF.\n\nCloses #10249\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10259 from davies/udf_null.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/b1b4ee7f3541d92c8bc2b0b4fdadf46cfdb09504"},{"sha":"518ab5101073ee35d62e33c8f7281a1e6342101e","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-10991][ML] logistic regression training summary handle empty prediction col\n\nLogisticRegression training summary should still function if the predictionCol is set to an empty string or otherwise unset (related too https://issues.apache.org/jira/browse/SPARK-9718 )\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #9037 from holdenk/SPARK-10991-LogisticRegressionTrainingSummary-handle-empty-prediction-col.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/518ab5101073ee35d62e33c8f7281a1e6342101e"},{"sha":"c119a34d1e9e599e302acfda92e5de681086a19f","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12258] [SQL] passing null into ScalaUDF (follow-up)\n\nThis is a follow-up PR for #10259\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10266 from davies/null_udf2.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/c119a34d1e9e599e302acfda92e5de681086a19f"},{"sha":"0fb9825556dbbcc98d7eafe9ddea8676301e09bb","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-12146][SPARKR] SparkR jsonFile should support multiple input files\n\n* ```jsonFile``` should support multiple input files, such as:\n```R\njsonFile(sqlContext, c(“path1”, “path2”)) # character vector as arguments\njsonFile(sqlContext, “path1,path2”)\n```\n* Meanwhile, ```jsonFile``` has been deprecated by Spark SQL and will be removed at Spark 2.0. So we mark ```jsonFile``` deprecated and use ```read.json``` at SparkR side.\n* Replace all ```jsonFile``` with ```read.json``` at test_sparkSQL.R, but still keep jsonFile test case.\n* If this PR is accepted, we should also make almost the same change for ```parquetFile```.\n\ncc felixcheung sun-rui shivaram\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #10145 from yanboliang/spark-12146.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/0fb9825556dbbcc98d7eafe9ddea8676301e09bb"},{"sha":"aa305dcaf5b4148aba9e669e081d0b9235f50857","author":{"email":"6723bdb1c0c6de39646bce6cc5d8a057d4f109d5@gmail.com","name":"anabranch"},"message":"[SPARK-11964][DOCS][ML] Add in Pipeline Import/Export Documentation\n\nAdding in Pipeline Import and Export Documentation.\n\nAuthor: anabranch <wac.chambers@gmail.com>\nAuthor: Bill Chambers <wchambers@ischool.berkeley.edu>\n\nCloses #10179 from anabranch/master.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/aa305dcaf5b4148aba9e669e081d0b9235f50857"},{"sha":"713e6959d21d24382ef99bbd7e9da751a7ed388c","author":{"email":"0a5ae72d3567a55d9fadf2c4c5fb00f8ec263cad@gmail.com","name":"proflin"},"message":"[SPARK-12273][STREAMING] Make Spark Streaming web UI list Receivers in order\n\nCurrently the Streaming web UI does NOT list Receivers in order; however, it seems more convenient for the users if Receivers are listed in order.\n\n![spark-12273](https://cloud.githubusercontent.com/assets/15843379/11736602/0bb7f7a8-a00b-11e5-8e86-96ba9297fb12.png)\n\nAuthor: proflin <proflin.me@gmail.com>\n\nCloses #10264 from proflin/Spark-12273.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/713e6959d21d24382ef99bbd7e9da751a7ed388c"},{"sha":"1b8220387e6903564f765fabb54be0420c3e99d7","author":{"email":"3c26cc6a278bab57a956b43cbc3c2947e24baa4a@us.ibm.com","name":"Mike Dusenberry"},"message":"[SPARK-11497][MLLIB][PYTHON] PySpark RowMatrix Constructor Has Type Erasure Issue\n\nAs noted in PR #9441, implementing `tallSkinnyQR` uncovered a bug with our PySpark `RowMatrix` constructor.  As discussed on the dev list [here](http://apache-spark-developers-list.1001551.n3.nabble.com/K-Means-And-Class-Tags-td10038.html), there appears to be an issue with type erasure with RDDs coming from Java, and by extension from PySpark.  Although we are attempting to construct a `RowMatrix` from an `RDD[Vector]` in [PythonMLlibAPI](https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala#L1115), the `Vector` type is erased, resulting in an `RDD[Object]`.  Thus, when calling Scala's `tallSkinnyQR` from PySpark, we get a Java `ClassCastException` in which an `Object` cannot be cast to a Spark `Vector`.  As noted in the aforementioned dev list thread, this issue was also encountered with `DecisionTrees`, and the fix involved an explicit `retag` of the RDD with a `Vector` type.  `IndexedRowMatrix` and `CoordinateMatrix` do not appear to have this issue likely due to their related helper functions in `PythonMLlibAPI` creating the RDDs explicitly from DataFrames with pattern matching, thus preserving the types.\n\nThis PR currently contains that retagging fix applied to the `createRowMatrix` helper function in `PythonMLlibAPI`.  This PR blocks #9441, so once this is merged, the other can be rebased.\n\ncc holdenk\n\nAuthor: Mike Dusenberry <mwdusenb@us.ibm.com>\n\nCloses #9458 from dusenberrymw/SPARK-11497_PySpark_RowMatrix_Constructor_Has_Type_Erasure_Issue.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/1b8220387e6903564f765fabb54be0420c3e99d7"},{"sha":"aea676ca2d07c72b1a752e9308c961118e5bfc3c","author":{"email":"07c870ffe79a3d57c30fc67475f42d361ec88a03@gmail.com","name":"BenFradet"},"message":"[SPARK-12217][ML] Document invalid handling for StringIndexer\n\nAdded a paragraph regarding StringIndexer#setHandleInvalid to the ml-features documentation.\n\nI wonder if I should also add a snippet to the code example, input welcome.\n\nAuthor: BenFradet <benjamin.fradet@gmail.com>\n\nCloses #10257 from BenFradet/SPARK-12217.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/aea676ca2d07c72b1a752e9308c961118e5bfc3c"}]}