{"push_id":857223452,"size":27,"distinct_size":27,"ref":"refs/heads/branch-1.6","head":"696d4a52d8ee5c1c736ce470ac87255fe58e78c3","before":"37c59f0ba5c6f5effde2212ca2952058a2fb9ea1","commits":[{"sha":"83b957a35bd4cd260a48af8cd602a2a67d08b4b9","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-11537] [SQL] fix negative hours/minutes/seconds\n\nCurrently, if the Timestamp is before epoch (1970/01/01), the hours, minutes and seconds will be negative (also rounding up).\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #9502 from davies/neg_hour.\n\n(cherry picked from commit 07414afac9a100ede1dee5f3d45a657802c8bd2a)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/83b957a35bd4cd260a48af8cd602a2a67d08b4b9"},{"sha":"de03021ac9058e56481c227028cdd4635c803713","author":{"email":"bc4a65f227ddcd81bb538c67e810a3e11696e8bf@gmail.com","name":"Yu ISHIKAWA"},"message":"[SPARK-11514][ML] Pass random seed to spark.ml DecisionTree*\n\ncc jkbradley\n\nAuthor: Yu ISHIKAWA <yuu.ishikawa@gmail.com>\n\nCloses #9486 from yu-iskw/SPARK-11514.\n\n(cherry picked from commit 8fa8c8375d7015a0332aa9ee613d7c6b6d62bae7)\nSigned-off-by: Joseph K. Bradley <joseph@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/de03021ac9058e56481c227028cdd4635c803713"},{"sha":"b655117d02b7429b6ca245812ab77458c022717f","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"Revert \"[SPARK-11469][SQL] Allow users to define nondeterministic udfs.\"\n\nThis reverts commit 9cf56c96b7d02a14175d40b336da14c2e1c88339.\n\nConflicts:\n\tproject/MimaExcludes.scala","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/b655117d02b7429b6ca245812ab77458c022717f"},{"sha":"1b43dd391b2233c1b062df886628eb6fb92b2936","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-7542][SQL] Support off-heap index/sort buffer\n\nThis brings the support of off-heap memory for array inside BytesToBytesMap and InMemorySorter, then we could allocate all the memory from off-heap for execution.\n\nCloses #8068\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #9477 from davies/unsafe_timsort.\n\n(cherry picked from commit eec74ba8bde7f9446cc38e687bda103e85669d35)\nSigned-off-by: Josh Rosen <joshrosen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/1b43dd391b2233c1b062df886628eb6fb92b2936"},{"sha":"1cfad7d557e52c0428230f2c32b054670cb37cf3","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-11528] [SQL] Typed aggregations for Datasets\n\nThis PR adds the ability to do typed SQL aggregations.  We will likely also want to provide an interface to allow users to do aggregations on objects, but this is deferred to another PR.\n\n```scala\nval ds = Seq((\"a\", 10), (\"a\", 20), (\"b\", 1), (\"b\", 2), (\"c\", 1)).toDS()\nds.groupBy(_._1).agg(sum(\"_2\").as[Int]).collect()\n\nres0: Array((\"a\", 30), (\"b\", 3), (\"c\", 1))\n```\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #9499 from marmbrus/dataset-agg.\n\n(cherry picked from commit 363a476c3fefb0263e63fd24df0b2779a64f79ec)\nSigned-off-by: Michael Armbrust <michael@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/1cfad7d557e52c0428230f2c32b054670cb37cf3"},{"sha":"0a430f04eef3445fb0095adc806d91759eea5d32","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-11511][STREAMING] Fix NPE when an InputDStream is not used\n\nJust ignored `InputDStream`s that have null `rememberDuration` in `DStreamGraph.getMaxInputStreamRememberDuration`.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #9476 from zsxwing/SPARK-11511.\n\n(cherry picked from commit cf69ce136590fea51843bc54f44f0f45c7d0ac36)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/0a430f04eef3445fb0095adc806d91759eea5d32"},{"sha":"d69bc9e4754d9638ca525a0880118bcf5ab2ad05","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-9162] [SQL] Implement code generation for ScalaUDF\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-9162\n\nCurrently ScalaUDF extends CodegenFallback and doesn't provide code generation implementation. This path implements code generation for ScalaUDF.\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #9270 from viirya/scalaudf-codegen.\n\n(cherry picked from commit 574141a29835ce78d68c97bb54336cf4fd3c39d3)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/d69bc9e4754d9638ca525a0880118bcf5ab2ad05"},{"sha":"fba48e7332b58169bda215c2a261ae9195c67744","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-10978][SQL][FOLLOW-UP] More comprehensive tests for PR #9399\n\nThis PR adds test cases that test various column pruning and filter push-down cases.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #9468 from liancheng/spark-10978.follow-up.\n\n(cherry picked from commit c048929c6a9f7ce57f384037cd6c0bf5751c447a)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/fba48e7332b58169bda215c2a261ae9195c67744"},{"sha":"089ba81d10630dfb046a71e32c1d1804511f050f","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-9858][SQL] Add an ExchangeCoordinator to estimate the number of post-shuffle partitions for aggregates and joins (follow-up)\n\nhttps://issues.apache.org/jira/browse/SPARK-9858\n\nThis PR is the follow-up work of https://github.com/apache/spark/pull/9276. It addresses JoshRosen's comments.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #9453 from yhuai/numReducer-followUp.\n\n(cherry picked from commit 8211aab0793cf64202b99be4f31bb8a9ae77050d)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/089ba81d10630dfb046a71e32c1d1804511f050f"},{"sha":"9d6238859e98651f32b9dd733a83c0e6f45978a7","author":{"email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com","name":"jerryshao"},"message":"[SPARK-11457][STREAMING][YARN] Fix incorrect AM proxy filter conf recovery from checkpoint\n\nCurrently Yarn AM proxy filter configuration is recovered from checkpoint file when Spark Streaming application is restarted, which will lead to some unwanted behaviors:\n\n1. Wrong RM address if RM is redeployed from failure.\n2. Wrong proxyBase, since app id is updated, old app id for proxyBase is wrong.\n\nSo instead of recovering from checkpoint file, these configurations should be reloaded each time when app started.\n\nThis problem only exists in Yarn cluster mode, for Yarn client mode, these configurations will be updated with RPC message `AddWebUIFilter`.\n\nPlease help to review tdas harishreedharan vanzin , thanks a lot.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #9412 from jerryshao/SPARK-11457.\n\n(cherry picked from commit 468ad0ae874d5cf55712ee976faf77f19c937ccb)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/9d6238859e98651f32b9dd733a83c0e6f45978a7"},{"sha":"051b2ca3a0aa18c5d805cbd183bca504865297c4","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-11538][BUILD] Force guava 14 in sbt build.\n\nsbt's version resolution code always picks the most recent version, and we\ndon't want that for guava.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #9508 from vanzin/SPARK-11538.\n\n(cherry picked from commit 5e31db70bb783656ba042863fcd3c223e17a8f81)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/051b2ca3a0aa18c5d805cbd183bca504865297c4"},{"sha":"a015f8141f4123d0389594e890d25b1a0f0adc71","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11540][SQL] API audit for QueryExecutionListener.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9509 from rxin/SPARK-11540.\n\n(cherry picked from commit 3cc2c053b5d68c747a30bd58cf388b87b1922f13)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/a015f8141f4123d0389594e890d25b1a0f0adc71"},{"sha":"641cdfd869cf6096274f55adcccdf01caa9752b5","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11541][SQL] Break JdbcDialects.scala into multiple files and mark various dialects as private.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9511 from rxin/SPARK-11541.\n\n(cherry picked from commit bc5d6c03893a9bd340d6b94d3550e25648412241)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/641cdfd869cf6096274f55adcccdf01caa9752b5"},{"sha":"e8649995418fb9c3a172181f0d0fa6d49ba2f674","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11453][SQL][FOLLOW-UP] remove DecimalLit\n\nA cleanup for https://github.com/apache/spark/pull/9085.\n\nThe `DecimalLit` is very similar to `FloatLit`, we can just keep one of them.\nAlso added low level unit test at `SqlParserSuite`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9482 from cloud-fan/parser.\n\n(cherry picked from commit 253e87e8ab8717ffef40a6d0d376b1add155ef90)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/e8649995418fb9c3a172181f0d0fa6d49ba2f674"},{"sha":"7755b50b4352432c1ab9d600dd4f4b52727bfa40","author":{"email":"b54fbb0e17aa598fdc04bb3d79ba76b5f0737666@cloudera.com","name":"Imran Rashid"},"message":"[SPARK-10116][CORE] XORShiftRandom.hashSeed is random in high bits\n\nhttps://issues.apache.org/jira/browse/SPARK-10116\n\nThis is really trivial, just happened to notice it -- if `XORShiftRandom.hashSeed` is really supposed to have random bits throughout (as the comment implies), it needs to do something for the conversion to `long`.\n\nmengxr mkolod\n\nAuthor: Imran Rashid <irashid@cloudera.com>\n\nCloses #8314 from squito/SPARK-10116.\n\n(cherry picked from commit 49f1a820372d1cba41f3f00d07eb5728f2ed6705)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/7755b50b4352432c1ab9d600dd4f4b52727bfa40"},{"sha":"efa1e4a25cd6f4311faf9fc6c551ef1c71e2227b","author":{"email":"679c4ded150c6b9a8052807f108ce836961fed8d@questtec.nl","name":"Herman van Hovell"},"message":"[SPARK-11450] [SQL] Add Unsafe Row processing to Expand\n\nThis PR enables the Expand operator to process and produce Unsafe Rows.\n\nAuthor: Herman van Hovell <hvanhovell@questtec.nl>\n\nCloses #9414 from hvanhovell/SPARK-11450.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/efa1e4a25cd6f4311faf9fc6c551ef1c71e2227b"},{"sha":"52e921c7cc6ba6dddb923221c42f52eb4cd98f0f","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11561][SQL] Rename text data source's column name to value.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9527 from rxin/SPARK-11561.\n\n(cherry picked from commit 3a652f691b220fada0286f8d0a562c5657973d4d)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/52e921c7cc6ba6dddb923221c42f52eb4cd98f0f"},{"sha":"e7e3bfda315eb7809c8bdffa61a30b70a680611c","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-11217][ML] save/load for non-meta estimators and transformers\n\nThis PR implements the default save/load for non-meta estimators and transformers using the JSON serialization of param values. The saved metadata includes:\n\n* class name\n* uid\n* timestamp\n* paramMap\n\nThe save/load interface is similar to DataFrames. We use the current active context by default, which should be sufficient for most use cases.\n\n~~~scala\ninstance.save(\"path\")\ninstance.write.context(sqlContext).overwrite().save(\"path\")\n\nInstance.load(\"path\")\n~~~\n\nThe param handling is different from the design doc. We didn't save default and user-set params separately, and when we load it back, all parameters are user-set. This does cause issues. But it also cause other issues if we modify the default params.\n\nTODOs:\n\n* [x] Java test\n* [ ] a follow-up PR to implement default save/load for all non-meta estimators and transformers\n\ncc jkbradley\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #9454 from mengxr/SPARK-11217.\n\n(cherry picked from commit c447c9d54603890db7399fb80adc9fae40b71f64)\nSigned-off-by: Joseph K. Bradley <joseph@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/e7e3bfda315eb7809c8bdffa61a30b70a680611c"},{"sha":"b58f1ce5b768f77b86626767f9366c00d2232b04","author":{"email":"a1a405b82800b492a7268cba6bc58270d2b527de@staydecay.corp.gq1.yahoo.com","name":"Thomas Graves"},"message":"[SPARK-11555] spark on yarn spark-class --num-workers doesn't work\n\nI tested the various options with both spark-submit and spark-class of specifying number of executors in both client and cluster mode where it applied.\n\n--num-workers, --num-executors, spark.executor.instances, SPARK_EXECUTOR_INSTANCES, default nothing supplied\n\nAuthor: Thomas Graves <tgraves@staydecay.corp.gq1.yahoo.com>\n\nCloses #9523 from tgravescs/SPARK-11555.\n\n(cherry picked from commit f6680cdc5d2912dea9768ef5c3e2cc101b06daf8)\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/b58f1ce5b768f77b86626767f9366c00d2232b04"},{"sha":"02748c953cdc85fda054e366b095225f906b4548","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11269][SQL] Java API support & test cases for Dataset\n\nThis simply brings https://github.com/apache/spark/pull/9358 up-to-date.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9528 from rxin/dataset-java.\n\n(cherry picked from commit 7e9a9e603abce8689938bdd62d04b29299644aa4)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/02748c953cdc85fda054e366b095225f906b4548"}]}