{"push_id":791844086,"size":45,"distinct_size":45,"ref":"refs/heads/master","head":"32407bfd2bdbf84d65cacfa7554dae6a2332bc37","before":"71da1633c4dcc4e748fbe3b3236af90032b695ae","commits":[{"sha":"45de518742446ddfbd4816c9d0f8501139f9bc2d","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-9730] [SQL] Add Full Outer Join support for SortMergeJoin\n\nThis PR is based on #8383 , thanks to viirya\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-9730\n\nThis patch adds the Full Outer Join support for SortMergeJoin. A new class SortMergeFullJoinScanner is added to scan rows from left and right iterators. FullOuterIterator is simply a wrapper of type RowIterator to consume joined rows from SortMergeFullJoinScanner.\n\nCloses #8383\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #8579 from davies/smj_fullouter.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/45de518742446ddfbd4816c9d0f8501139f9bc2d"},{"sha":"56a0fe5c6e4ae2929c48fae2d6225558d020e5f9","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-9772] [PYSPARK] [ML] Add Python API for ml.feature.VectorSlicer\n\nAdd Python API for ml.feature.VectorSlicer.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8102 from yanboliang/SPARK-9772.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/56a0fe5c6e4ae2929c48fae2d6225558d020e5f9"},{"sha":"1dc7548c598c4eb4ecc7d5bb8962a735bbd2c0f7","author":{"email":"593c8defb1577d6e2c84e1e8a9c93cdf8b83b8d0@gmail.com","name":"Sean Paradiso"},"message":"[MINOR] [MLLIB] [ML] [DOC] fixed typo: label for negative result should be 0.0 (original: 1.0)\n\nSmall typo in the example for `LabelledPoint` in the MLLib docs.\n\nAuthor: Sean Paradiso <seanparadiso@gmail.com>\n\nCloses #8680 from sparadiso/docs_mllib_smalltypo.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/1dc7548c598c4eb4ecc7d5bb8962a735bbd2c0f7"},{"sha":"48817cc111a9705f40b7c842315eee24291c2198","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-10497] [BUILD] [TRIVIAL] Handle both locations for JIRAError with python-jira\n\nLocation of JIRAError has moved between old and new versions of python-jira package.\nLonger term it probably makes sense to pin to specific versions (as mentioned in https://issues.apache.org/jira/browse/SPARK-10498 ) but for now, making release tools works with both new and old versions of python-jira.\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #8661 from holdenk/SPARK-10497-release-utils-does-not-work-with-new-jira-python.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/48817cc111a9705f40b7c842315eee24291c2198"},{"sha":"4f1daa1ef6b36440962f3c8faea3928599e33784","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-10065] [SQL] avoid the extra copy when generate unsafe array\n\nThe reason for this extra copy is that we iterate the array twice: calculate elements data size and copy elements to array buffer.\n\nA simple solution is to follow `createCodeForStruct`, we can dynamically grow the buffer when needed and thus don't need to know the data size ahead.\n\nThis PR also include some typo and style fixes, and did some minor refactor to make sure `input.primitive` is always variable name not code when generate unsafe code.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #8496 from cloud-fan/avoid-copy.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/4f1daa1ef6b36440962f3c8faea3928599e33784"},{"sha":"f892d927d7246856dd3ea617b2942873359454bc","author":{"email":"808d31295a27ded55339ed818ba0b1912ca15edb@guavus.com","name":"Yash Datta"},"message":"[SPARK-7142] [SQL] Minor enhancement to BooleanSimplification Optimizer rule\n\nUse these in the optimizer as well:\n\n            A and (not(A) or B) => A and B\n            not(A and B) => not(A) or not(B)\n            not(A or B) => not(A) and not(B)\n\nAuthor: Yash Datta <Yash.Datta@guavus.com>\n\nCloses #5700 from saucam/bool_simp.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/f892d927d7246856dd3ea617b2942873359454bc"},{"sha":"49da38e5f728e05e8e929c4dcd37145ba060151d","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-10301] [SPARK-10428] [SQL] Addresses comments of PR #8583 and #8509 for master\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #8670 from liancheng/spark-10301/address-pr-comments.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/49da38e5f728e05e8e929c4dcd37145ba060151d"},{"sha":"e04811137680f937669cdcc78771227aeb7cd849","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-10466] [SQL] UnsafeRow SerDe exception with data spill\n\nData Spill with UnsafeRow causes assert failure.\n\n```\njava.lang.AssertionError: assertion failed\n\tat scala.Predef$.assert(Predef.scala:165)\n\tat org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$2.writeKey(UnsafeRowSerializer.scala:75)\n\tat org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:180)\n\tat org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$2$$anonfun$apply$1.apply(ExternalSorter.scala:688)\n\tat org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$2$$anonfun$apply$1.apply(ExternalSorter.scala:687)\n\tat scala.collection.Iterator$class.foreach(Iterator.scala:727)\n\tat scala.collection.AbstractIterator.foreach(Iterator.scala:1157)\n\tat org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$2.apply(ExternalSorter.scala:687)\n\tat org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$2.apply(ExternalSorter.scala:683)\n\tat scala.collection.Iterator$class.foreach(Iterator.scala:727)\n\tat scala.collection.AbstractIterator.foreach(Iterator.scala:1157)\n\tat org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:683)\n\tat org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:80)\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:88)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\n```\n\nTo reproduce that with code (thanks andrewor14):\n```scala\nbin/spark-shell --master local\n  --conf spark.shuffle.memoryFraction=0.005\n  --conf spark.shuffle.sort.bypassMergeThreshold=0\n\nsc.parallelize(1 to 2 * 1000 * 1000, 10)\n  .map { i => (i, i) }.toDF(\"a\", \"b\").groupBy(\"b\").avg().count()\n```\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #8635 from chenghao-intel/unsafe_spill.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/e04811137680f937669cdcc78771227aeb7cd849"},{"sha":"a76bde9dae54c4641e21f3c1ceb4870e3dc91881","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-10469] [DOC] Try and document the three options\n\nFrom JIRA:\nAdd documentation for tungsten-sort.\nFrom the mailing list \"I saw a new \"spark.shuffle.manager=tungsten-sort\" implemented in\nhttps://issues.apache.org/jira/browse/SPARK-7081, but it can't be found its\ncorresponding description in\nhttp://people.apache.org/~pwendell/spark-releases/spark-1.5.0-rc3-docs/configuration.html(Currenlty\nthere are only 'sort' and 'hash' two options).\"\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #8638 from holdenk/SPARK-10469-document-tungsten-sort.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/a76bde9dae54c4641e21f3c1ceb4870e3dc91881"},{"sha":"af3bc59d1f5d9d952c2d7ad1af599c49f1dbdaf0","author":{"email":"2e9bf58ec2f93170404b3a28538fb27400f699db@palantir.com","name":"mcheah"},"message":"[SPARK-8167] Make tasks that fail from YARN preemption not fail job\n\nThe architecture is that, in YARN mode, if the driver detects that an executor has disconnected, it asks the ApplicationMaster why the executor died. If the ApplicationMaster is aware that the executor died because of preemption, all tasks associated with that executor are not marked as failed. The executor\nis still removed from the driver's list of available executors, however.\n\nThere's a few open questions:\n1. Should standalone mode have a similar \"get executor loss reason\" as well? I localized this change as much as possible to affect only YARN, but there could be a valid case to differentiate executor losses in standalone mode as well.\n2. I make a pretty strong assumption in YarnAllocator that getExecutorLossReason(executorId) will only be called once per executor id; I do this so that I can remove the metadata from the in-memory map to avoid object accumulation. It's not clear if I'm being overly zealous to save space, however.\n\ncc vanzin specifically for review because it collided with some earlier YARN scheduling work.\ncc JoshRosen because it's similar to output commit coordination we did in the past\ncc andrewor14 for our discussion on how to get executor exit codes and loss reasons\n\nAuthor: mcheah <mcheah@palantir.com>\n\nCloses #8007 from mccheah/feature/preemption-handling.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/af3bc59d1f5d9d952c2d7ad1af599c49f1dbdaf0"},{"sha":"f0562e8cdbab7ce40f3186da98595312252f8b5c","author":{"email":"53fdba43521c8d2c2b4ff3a6f81e9f27ad86e7ac@gmail.com","name":"Iulian Dragos"},"message":"[SPARK-6350] [MESOS] Fine-grained mode scheduler respects mesosExecutor.cores\n\nThis is a regression introduced in #4960, this commit fixes it and adds a test.\n\ntnachen andrewor14 please review, this should be an easy one.\n\nAuthor: Iulian Dragos <jaguarul@gmail.com>\n\nCloses #8653 from dragos/issue/mesos/fine-grained-maxExecutorCores.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/f0562e8cdbab7ce40f3186da98595312252f8b5c"},{"sha":"a5ef2d0600d5e23ca05fabc1005bb81e5ada0727","author":{"email":"7ba0da326b36621a0fa584ce084ec977fe1343f7@gmail.com","name":"Akash Mishra"},"message":"[SPARK-10514] [MESOS] waiting for min no of total cores acquired by Spark by implementing the sufficientResourcesRegistered method\n\nspark.scheduler.minRegisteredResourcesRatio configuration parameter works for YARN mode but not for Mesos Coarse grained mode.\n\nIf the parameter specified default value of 0 will be set for spark.scheduler.minRegisteredResourcesRatio in base class and this method will always return true.\n\nThere are no existing test for YARN mode too. Hence not added test for the same.\n\nAuthor: Akash Mishra <akash.mishra20@gmail.com>\n\nCloses #8672 from SleepyThread/master.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/a5ef2d0600d5e23ca05fabc1005bb81e5ada0727"},{"sha":"d88abb7e212fb55f9b0398a0f76a753c86b85cf1","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-9990] [SQL] Create local hash join operator\n\nThis PR includes the following changes:\n- Add SQLConf to LocalNode\n- Add HashJoinNode\n- Add ConvertToUnsafeNode and ConvertToSafeNode.scala to test unsafe hash join.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #8535 from zsxwing/SPARK-9990.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/d88abb7e212fb55f9b0398a0f76a753c86b85cf1"},{"sha":"45e3be5c138d983f40f619735d60bf7eb78c9bf0","author":{"email":"3f27eab2afd49386f0ac811afb0bc245aa92c0fa@intel.com","name":"Sun Rui"},"message":"[SPARK-10049] [SPARKR] Support collecting data of ArraryType in DataFrame.\n\nthis PR :\n1.  Enhance reflection in RBackend. Automatically matching a Java array to Scala Seq when finding methods. Util functions like seq(), listToSeq() in R side can be removed, as they will conflict with the Serde logic that transferrs a Scala seq to R side.\n\n2.  Enhance the SerDe to support transferring  a Scala seq to R side. Data of ArrayType in DataFrame\nafter collection is observed to be of Scala Seq type.\n\n3.  Support ArrayType in createDataFrame().\n\nAuthor: Sun Rui <rui.sun@intel.com>\n\nCloses #8458 from sun-rui/SPARK-10049.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/45e3be5c138d983f40f619735d60bf7eb78c9bf0"},{"sha":"3db72554be3f13478ccd5915e746491982163298","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-10443] [SQL] Refactor SortMergeOuterJoin to reduce duplication\n\n`LeftOutputIterator` and `RightOutputIterator` are symmetrically identical and can share a lot of code. If someone makes a change in one but forgets to do the same thing in the other we'll end up with inconsistent behavior. This patch also adds inline comments to clarify the intention of the code.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #8596 from andrewor14/smoj-cleanup.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/3db72554be3f13478ccd5915e746491982163298"},{"sha":"4204757714b36364611fb63bc008cf90fc53d8df","author":{"email":"ad63b47dfac1951bde2b661ba9fbbe123f686bee@cs.berkeley.edu","name":"Shivaram Venkataraman"},"message":"Add 1.5 to master branch EC2 scripts\n\nThis change brings it to par with `branch-1.5` (and 1.5.0 release)\n\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\n\nCloses #8704 from shivaram/ec2-1.5-update.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/4204757714b36364611fb63bc008cf90fc53d8df"},{"sha":"89562a172fd3efa032f60714d600407c6cfe2c2f","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-7544] [SQL] [PySpark] pyspark.sql.types.Row implements __getitem__\n\npyspark.sql.types.Row implements ```__getitem__```\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8333 from yanboliang/spark-7544.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/89562a172fd3efa032f60714d600407c6cfe2c2f"},{"sha":"0eabea8a058ad60411c1384930ba12c1c638f5f1","author":{"email":"7841830342f748a54814a395669113bed0a8f742@cs.berkeley.edu","name":"Matt Massie"},"message":"[SPARK-9043] Serialize key, value and combiner classes in ShuffleDependency\n\nShuffleManager implementations are currently not given type information for\nthe key, value and combiner classes. Serialization of shuffle objects relies\non objects being JavaSerializable, with methods defined for reading/writing\nthe object or, alternatively, serialization via Kryo which uses reflection.\n\nSerialization systems like Avro, Thrift and Protobuf generate classes with\nzero argument constructors and explicit schema information\n(e.g. IndexedRecords in Avro have get, put and getSchema methods).\n\nBy serializing the key, value and combiner class names in ShuffleDependency,\nshuffle implementations will have access to schema information when\nregisterShuffle() is called.\n\nAuthor: Matt Massie <massie@cs.berkeley.edu>\n\nCloses #7403 from massie/shuffle-classtags.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/0eabea8a058ad60411c1384930ba12c1c638f5f1"},{"sha":"339a527141984bfb182862b0987d3c4690c9ede1","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-10023] [ML] [PySpark] Unified DecisionTreeParams checkpointInterval between Scala and Python API.\n\n\"checkpointInterval\" is member of DecisionTreeParams in Scala API which is inconsistency with Python API, we should unified them.\n```\nmember of DecisionTreeParams <-> Scala API\nshared param for all ML Transformer/Estimator <-> Python API\n```\nProposal:\n\"checkpointInterval\" is also used by ALS, so we make it shared params at Scala.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8528 from yanboliang/spark-10023.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/339a527141984bfb182862b0987d3c4690c9ede1"},{"sha":"a140dd77c62255d6f7f6817a2517d47feb8540d4","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-10027] [ML] [PySpark] Add Python API missing methods for ml.feature\n\nMissing method of ml.feature are listed here:\n```StringIndexer``` lacks of parameter ```handleInvalid```.\n```StringIndexerModel``` lacks of method ```labels```.\n```VectorIndexerModel``` lacks of methods ```numFeatures``` and ```categoryMaps```.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8313 from yanboliang/spark-10027.","distinct":true,"url":"https://api.github.com/repos/noel-smith/spark/commits/a140dd77c62255d6f7f6817a2517d47feb8540d4"}]}