{"push_id":841690440,"size":464,"distinct_size":464,"ref":"refs/heads/master","head":"d4c397a64af4cec899fdaa3e617ed20333cc567d","before":"ae74c3fa84885dac18f39368e6dda48a2f7a25a5","commits":[{"sha":"820913f554bef610d07ca2dadaead657f916ae63","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-10071] [STREAMING] Output a warning when writing QueueInputDStream and throw a better exception when reading QueueInputDStream\n\nOutput a warning when serializing QueueInputDStream rather than throwing an exception to allow unit tests use it. Moreover, this PR also throws an better exception when deserializing QueueInputDStream to make the user find out the problem easily. The previous exception is hard to understand: https://issues.apache.org/jira/browse/SPARK-8553\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #8624 from zsxwing/SPARK-10071 and squashes the following commits:\n\n847cfa8 [zsxwing] Output a warning when writing QueueInputDStream and throw a better exception when reading QueueInputDStream","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/820913f554bef610d07ca2dadaead657f916ae63"},{"sha":"52fe32f6ac7a04fa9b4478fda1307c5b0c61c4a2","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-9834] [MLLIB] implement weighted least squares via normal equation\n\nThe goal of this PR is to have a weighted least squares implementation that takes the normal equation approach, and hence to be able to provide R-like summary statistics and support IRLS (used by GLMs). The tests match R's lm and glmnet.\n\nThere are couple TODOs that can be addressed in future PRs:\n* consolidate summary statistics aggregators\n* move `dspr` to `BLAS`\n* etc\n\nIt would be nice to have this merged first because it blocks couple other features.\n\ndbtsai\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #8588 from mengxr/SPARK-9834.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/52fe32f6ac7a04fa9b4478fda1307c5b0c61c4a2"},{"sha":"a1573489a37def97b7c26b798898ffbbdc4defa8","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-10464] [MLLIB] Add WeibullGenerator for RandomDataGenerator\n\nAdd WeibullGenerator for RandomDataGenerator.\n#8611 need use WeibullGenerator to generate random data based on Weibull distribution.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8622 from yanboliang/spark-10464.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/a1573489a37def97b7c26b798898ffbbdc4defa8"},{"sha":"3a11e50e21ececbec9708eb487b08196f195cd87","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-10373] [PYSPARK] move @since into pyspark from sql\n\ncc mengxr\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #8657 from davies/move_since.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/3a11e50e21ececbec9708eb487b08196f195cd87"},{"sha":"0e2f2163314972f6be18e3453c64314d1bee7bb9","author":{"email":"1d6e1cf70ec6f9ab28d3ea4b27a49a77654d370e@noelsmith.com","name":"noelsmith"},"message":"[SPARK-10094] Pyspark ML Feature transformers marked as experimental\n\nModified class-level docstrings to mark all feature transformers in pyspark.ml as experimental.\n\nAuthor: noelsmith <mail@noelsmith.com>\n\nCloses #8623 from noel-smith/SPARK-10094-mark-pyspark-ml-trans-exp.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/0e2f2163314972f6be18e3453c64314d1bee7bb9"},{"sha":"2f6fd5256c6650868916a3eefaa0beb091187cbb","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-9654] [ML] [PYSPARK] Add IndexToString to PySpark\n\nAdds IndexToString to PySpark.\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #7976 from holdenk/SPARK-9654-add-string-indexer-inverse-in-pyspark.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/2f6fd5256c6650868916a3eefaa0beb091187cbb"},{"sha":"91a577d2778ab5946f0c40cb80c89de24e3d10e8","author":{"email":"66332c93c294b372ab1bdd5acb9b89a538e2d2f2@gmail.com","name":"Yuhao Yang"},"message":"[SPARK-10249] [ML] [DOC] Add Python Code Example to StopWordsRemover User Guide\n\njira: https://issues.apache.org/jira/browse/SPARK-10249\n\nupdate user guide since python support added.\n\nAuthor: Yuhao Yang <hhbyyh@gmail.com>\n\nCloses #8620 from hhbyyh/swPyDocExample.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/91a577d2778ab5946f0c40cb80c89de24e3d10e8"},{"sha":"c1bc4f439f54625c01a585691e5293cd9961eb0c","author":{"email":"caac4775dfdb0d1258c8ff3f66efc735f5034de8@typesafe.com","name":"Luc Bourlier"},"message":"[SPARK-10227] fatal warnings with sbt on Scala 2.11\n\nThe bulk of the changes are on `transient` annotation on class parameter. Often the compiler doesn't generate a field for this parameters, so the the transient annotation would be unnecessary.\nBut if the class parameter are used in methods, then fields are created. So it is safer to keep the annotations.\n\nThe remainder are some potential bugs, and deprecated syntax.\n\nAuthor: Luc Bourlier <luc.bourlier@typesafe.com>\n\nCloses #8433 from skyluc/issue/sbt-2.11.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/c1bc4f439f54625c01a585691e5293cd9961eb0c"},{"sha":"2ddeb63126d26149eda197e85b7b26ef16a6e97c","author":{"email":"bb02c6aa60d4a0a091a9679b63db9e2367e07b50@me.com","name":"lewuathe"},"message":"[SPARK-10117] [MLLIB] Implement SQL data source API for reading LIBSVM data\n\nIt is convenient to implement data source API for LIBSVM format to have a better integration with DataFrames and ML pipeline API.\n\nTwo option is implemented.\n* `numFeatures`: Specify the dimension of features vector\n* `featuresType`: Specify the type of output vector. `sparse` is default.\n\nAuthor: lewuathe <lewuathe@me.com>\n\nCloses #8537 from Lewuathe/SPARK-10117 and squashes the following commits:\n\n986999d [lewuathe] Change unit test phrase\n11d513f [lewuathe] Fix some reviews\n21600a4 [lewuathe] Merge branch 'master' into SPARK-10117\n9ce63c7 [lewuathe] Rewrite service loader file\n1fdd2df [lewuathe] Merge branch 'SPARK-10117' of github.com:Lewuathe/spark into SPARK-10117\nba3657c [lewuathe] Merge branch 'master' into SPARK-10117\n0ea1c1c [lewuathe] LibSVMRelation is registered into META-INF\n4f40891 [lewuathe] Improve test suites\n5ab62ab [lewuathe] Merge branch 'master' into SPARK-10117\n8660d0e [lewuathe] Fix Java unit test\nb56a948 [lewuathe] Merge branch 'master' into SPARK-10117\n2c12894 [lewuathe] Remove unnecessary tag\n7d693c2 [lewuathe] Resolv conflict\n62010af [lewuathe] Merge branch 'master' into SPARK-10117\na97ee97 [lewuathe] Fix some points\naef9564 [lewuathe] Fix\n70ee4dd [lewuathe] Add Java test\n3fd8dce [lewuathe] [SPARK-10117] Implement SQL data source API for reading LIBSVM data\n40d3027 [lewuathe] Add Java test\n7056d4a [lewuathe] Merge branch 'master' into SPARK-10117\n99accaa [lewuathe] [SPARK-10117] Implement SQL data source API for reading LIBSVM data","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/2ddeb63126d26149eda197e85b7b26ef16a6e97c"},{"sha":"c0052d8d09eebadadb5ed35ac512caaf73919551","author":{"email":"ced9a24e2e494bb60341a9f26a61a78e7a11abad@apache.org","name":"Jeff Zhang"},"message":"[SPARK-10481] [YARN] SPARK_PREPEND_CLASSES make spark-yarn related jar could n…\n\nThrow a more readable exception. Please help review. Thanks\n\nAuthor: Jeff Zhang <zjffdu@apache.org>\n\nCloses #8649 from zjffdu/SPARK-10481.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/c0052d8d09eebadadb5ed35ac512caaf73919551"},{"sha":"71da1633c4dcc4e748fbe3b3236af90032b695ae","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-10461] [SQL] make sure `input.primitive` is always variable name not code at `GenerateUnsafeProjection`\n\nWhen we generate unsafe code inside `createCodeForXXX`, we always assign the `input.primitive` to a temp variable in case `input.primitive` is expression code.\n\nThis PR did some refactor to make sure `input.primitive` is always variable name, and some other typo and style fixes.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #8613 from cloud-fan/minor.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/71da1633c4dcc4e748fbe3b3236af90032b695ae"},{"sha":"45de518742446ddfbd4816c9d0f8501139f9bc2d","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-9730] [SQL] Add Full Outer Join support for SortMergeJoin\n\nThis PR is based on #8383 , thanks to viirya\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-9730\n\nThis patch adds the Full Outer Join support for SortMergeJoin. A new class SortMergeFullJoinScanner is added to scan rows from left and right iterators. FullOuterIterator is simply a wrapper of type RowIterator to consume joined rows from SortMergeFullJoinScanner.\n\nCloses #8383\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #8579 from davies/smj_fullouter.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/45de518742446ddfbd4816c9d0f8501139f9bc2d"},{"sha":"56a0fe5c6e4ae2929c48fae2d6225558d020e5f9","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-9772] [PYSPARK] [ML] Add Python API for ml.feature.VectorSlicer\n\nAdd Python API for ml.feature.VectorSlicer.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8102 from yanboliang/SPARK-9772.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/56a0fe5c6e4ae2929c48fae2d6225558d020e5f9"},{"sha":"1dc7548c598c4eb4ecc7d5bb8962a735bbd2c0f7","author":{"email":"593c8defb1577d6e2c84e1e8a9c93cdf8b83b8d0@gmail.com","name":"Sean Paradiso"},"message":"[MINOR] [MLLIB] [ML] [DOC] fixed typo: label for negative result should be 0.0 (original: 1.0)\n\nSmall typo in the example for `LabelledPoint` in the MLLib docs.\n\nAuthor: Sean Paradiso <seanparadiso@gmail.com>\n\nCloses #8680 from sparadiso/docs_mllib_smalltypo.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/1dc7548c598c4eb4ecc7d5bb8962a735bbd2c0f7"},{"sha":"48817cc111a9705f40b7c842315eee24291c2198","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-10497] [BUILD] [TRIVIAL] Handle both locations for JIRAError with python-jira\n\nLocation of JIRAError has moved between old and new versions of python-jira package.\nLonger term it probably makes sense to pin to specific versions (as mentioned in https://issues.apache.org/jira/browse/SPARK-10498 ) but for now, making release tools works with both new and old versions of python-jira.\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #8661 from holdenk/SPARK-10497-release-utils-does-not-work-with-new-jira-python.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/48817cc111a9705f40b7c842315eee24291c2198"},{"sha":"4f1daa1ef6b36440962f3c8faea3928599e33784","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-10065] [SQL] avoid the extra copy when generate unsafe array\n\nThe reason for this extra copy is that we iterate the array twice: calculate elements data size and copy elements to array buffer.\n\nA simple solution is to follow `createCodeForStruct`, we can dynamically grow the buffer when needed and thus don't need to know the data size ahead.\n\nThis PR also include some typo and style fixes, and did some minor refactor to make sure `input.primitive` is always variable name not code when generate unsafe code.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #8496 from cloud-fan/avoid-copy.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/4f1daa1ef6b36440962f3c8faea3928599e33784"},{"sha":"f892d927d7246856dd3ea617b2942873359454bc","author":{"email":"808d31295a27ded55339ed818ba0b1912ca15edb@guavus.com","name":"Yash Datta"},"message":"[SPARK-7142] [SQL] Minor enhancement to BooleanSimplification Optimizer rule\n\nUse these in the optimizer as well:\n\n            A and (not(A) or B) => A and B\n            not(A and B) => not(A) or not(B)\n            not(A or B) => not(A) and not(B)\n\nAuthor: Yash Datta <Yash.Datta@guavus.com>\n\nCloses #5700 from saucam/bool_simp.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/f892d927d7246856dd3ea617b2942873359454bc"},{"sha":"49da38e5f728e05e8e929c4dcd37145ba060151d","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-10301] [SPARK-10428] [SQL] Addresses comments of PR #8583 and #8509 for master\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #8670 from liancheng/spark-10301/address-pr-comments.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/49da38e5f728e05e8e929c4dcd37145ba060151d"},{"sha":"e04811137680f937669cdcc78771227aeb7cd849","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-10466] [SQL] UnsafeRow SerDe exception with data spill\n\nData Spill with UnsafeRow causes assert failure.\n\n```\njava.lang.AssertionError: assertion failed\n\tat scala.Predef$.assert(Predef.scala:165)\n\tat org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$2.writeKey(UnsafeRowSerializer.scala:75)\n\tat org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:180)\n\tat org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$2$$anonfun$apply$1.apply(ExternalSorter.scala:688)\n\tat org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$2$$anonfun$apply$1.apply(ExternalSorter.scala:687)\n\tat scala.collection.Iterator$class.foreach(Iterator.scala:727)\n\tat scala.collection.AbstractIterator.foreach(Iterator.scala:1157)\n\tat org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$2.apply(ExternalSorter.scala:687)\n\tat org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$2.apply(ExternalSorter.scala:683)\n\tat scala.collection.Iterator$class.foreach(Iterator.scala:727)\n\tat scala.collection.AbstractIterator.foreach(Iterator.scala:1157)\n\tat org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:683)\n\tat org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:80)\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:88)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\n```\n\nTo reproduce that with code (thanks andrewor14):\n```scala\nbin/spark-shell --master local\n  --conf spark.shuffle.memoryFraction=0.005\n  --conf spark.shuffle.sort.bypassMergeThreshold=0\n\nsc.parallelize(1 to 2 * 1000 * 1000, 10)\n  .map { i => (i, i) }.toDF(\"a\", \"b\").groupBy(\"b\").avg().count()\n```\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #8635 from chenghao-intel/unsafe_spill.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/e04811137680f937669cdcc78771227aeb7cd849"},{"sha":"a76bde9dae54c4641e21f3c1ceb4870e3dc91881","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-10469] [DOC] Try and document the three options\n\nFrom JIRA:\nAdd documentation for tungsten-sort.\nFrom the mailing list \"I saw a new \"spark.shuffle.manager=tungsten-sort\" implemented in\nhttps://issues.apache.org/jira/browse/SPARK-7081, but it can't be found its\ncorresponding description in\nhttp://people.apache.org/~pwendell/spark-releases/spark-1.5.0-rc3-docs/configuration.html(Currenlty\nthere are only 'sort' and 'hash' two options).\"\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #8638 from holdenk/SPARK-10469-document-tungsten-sort.","distinct":true,"url":"https://api.github.com/repos/jostep/spark/commits/a76bde9dae54c4641e21f3c1ceb4870e3dc91881"}]}