{"push_id":887018906,"size":26,"distinct_size":26,"ref":"refs/heads/branch-1.6","head":"d79dd971d01b69f8065b802fb5a78023ca905c7c","before":"1aa39bdb136ace5dcd4f7fdf62b8f324c14261f2","commits":[{"sha":"add4e63116b5d7a9bfc0e08ad3b5750d40897602","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-11949][SQL] Set field nullable property for GroupingSets to get correct results for null values\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-11949\n\nThe result of cube plan uses incorrect schema. The schema of cube result should set nullable property to true because the grouping expressions will have null values.\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #10038 from viirya/fix-cube.\n\n(cherry picked from commit c87531b765f8934a9a6c0f673617e0abfa5e5f0e)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/add4e63116b5d7a9bfc0e08ad3b5750d40897602"},{"sha":"9b99b2b46c452ba396e922db5fc7eec02c45b158","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-12060][CORE] Avoid memory copy in JavaSerializerInstance.serialize\n\n`JavaSerializerInstance.serialize` uses `ByteArrayOutputStream.toByteArray` to get the serialized data. `ByteArrayOutputStream.toByteArray` needs to copy the content in the internal array to a new array. However, since the array will be converted to `ByteBuffer` at once, we can avoid the memory copy.\n\nThis PR added `ByteBufferOutputStream` to access the protected `buf` and convert it to a `ByteBuffer` directly.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #10051 from zsxwing/SPARK-12060.\n\n(cherry picked from commit 1401166576c7018c5f9c31e0a6703d5fb16ea339)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/9b99b2b46c452ba396e922db5fc7eec02c45b158"},{"sha":"6e3e3c648c4f74d9c1aabe767dbadfe47bd7e658","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-12068][SQL] use a single column in Dataset.groupBy and count will fail\n\nThe reason is that, for a single culumn `RowEncoder`(or a single field product encoder), when we use it as the encoder for grouping key, we should also combine the grouping attributes, although there is only one grouping attribute.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10059 from cloud-fan/bug.\n\n(cherry picked from commit 8ddc55f1d582cccc3ca135510b2ea776e889e481)\nSigned-off-by: Michael Armbrust <michael@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/6e3e3c648c4f74d9c1aabe767dbadfe47bd7e658"},{"sha":"74a2306763161fc04c9d3e7de186a6b31617faf4","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11856][SQL] add type cast if the real type is different but compatible with encoder schema\n\nWhen we build the `fromRowExpression` for an encoder, we set up a lot of \"unresolved\" stuff and lost the required data type, which may lead to runtime error if the real type doesn't match the encoder's schema.\nFor example, we build an encoder for `case class Data(a: Int, b: String)` and the real type is `[a: int, b: long]`, then we will hit runtime error and say that we can't construct class `Data` with int and long, because we lost the information that `b` should be a string.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9840 from cloud-fan/err-msg.\n\n(cherry picked from commit 9df24624afedd993a39ab46c8211ae153aedef1a)\nSigned-off-by: Michael Armbrust <michael@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/74a2306763161fc04c9d3e7de186a6b31617faf4"},{"sha":"88bbce00813acf23eef411ac354c35995ddf9e77","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11954][SQL] Encoder for JavaBeans\n\ncreate java version of `constructorFor` and `extractorFor` in `JavaTypeInference`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nThis patch had conflicts when merged, resolved by\nCommitter: Michael Armbrust <michael@databricks.com>\n\nCloses #9937 from cloud-fan/pojo.\n\n(cherry picked from commit fd95eeaf491809c6bb0f83d46b37b5e2eebbcbca)\nSigned-off-by: Michael Armbrust <michael@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/88bbce00813acf23eef411ac354c35995ddf9e77"},{"sha":"40769b48cd001b7ff8c301628dc1442e3dd946cd","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-11905][SQL] Support Persist/Cache and Unpersist in Dataset APIs\n\nPersist and Unpersist exist in both RDD and Dataframe APIs. I think they are still very critical in Dataset APIs. Not sure if my understanding is correct? If so, could you help me check if the implementation is acceptable?\n\nPlease provide your opinions. marmbrus rxin cloud-fan\n\nThank you very much!\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\nAuthor: xiaoli <lixiao1983@gmail.com>\nAuthor: Xiao Li <xiaoli@Xiaos-MacBook-Pro.local>\n\nCloses #9889 from gatorsmile/persistDS.\n\n(cherry picked from commit 0a7bca2da04aefff16f2513ec27a92e69ceb77f6)\nSigned-off-by: Michael Armbrust <michael@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/40769b48cd001b7ff8c301628dc1442e3dd946cd"},{"sha":"843a31afbdeea66449750f0ba8f676ef31d00726","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-12046][DOC] Fixes various ScalaDoc/JavaDoc issues\n\nThis PR backports PR #10039 to master\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #10063 from liancheng/spark-12046.doc-fix.master.\n\n(cherry picked from commit 69dbe6b40df35d488d4ee343098ac70d00bbdafb)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/843a31afbdeea66449750f0ba8f676ef31d00726"},{"sha":"99dc1335e2f635a067f9fa1e83a35bf9593bfc24","author":{"email":"28fa304e82c936e69e01cd36723ee4e2aa7e0d24@gmail.com","name":"woj-i"},"message":"[SPARK-11821] Propagate Kerberos keytab for all environments\n\nandrewor14 the same PR as in branch 1.5\nharishreedharan\n\nAuthor: woj-i <wojciechindyk@gmail.com>\n\nCloses #9859 from woj-i/master.\n\n(cherry picked from commit 6a8cf80cc8ef435ec46138fa57325bda5d68f3ce)\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/99dc1335e2f635a067f9fa1e83a35bf9593bfc24"},{"sha":"ab2a124c8eca6823ee016c9ecfbdbf4918fbcdd6","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-12065] Upgrade Tachyon from 0.8.1 to 0.8.2\n\nThis commit upgrades the Tachyon dependency from 0.8.1 to 0.8.2.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10054 from JoshRosen/upgrade-to-tachyon-0.8.2.\n\n(cherry picked from commit 34e7093c1131162b3aa05b65a19a633a0b5b633e)\nSigned-off-by: Josh Rosen <joshrosen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/ab2a124c8eca6823ee016c9ecfbdbf4918fbcdd6"},{"sha":"1cf9d3858c8a3a5796b64a9fbea22509f02d778a","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@databricks.com","name":"Nong Li"},"message":"[SPARK-12030] Fix Platform.copyMemory to handle overlapping regions.\n\nThis bug was exposed as memory corruption in Timsort which uses copyMemory to copy\nlarge regions that can overlap. The prior implementation did not handle this case\nhalf the time and always copied forward, resulting in the data being corrupt.\n\nAuthor: Nong Li <nong@databricks.com>\n\nCloses #10068 from nongli/spark-12030.\n\n(cherry picked from commit 2cef1cdfbb5393270ae83179b6a4e50c3cbf9e93)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/1cf9d3858c8a3a5796b64a9fbea22509f02d778a"},{"sha":"81db8d086bbfe72caa0c45a395ebcaed80b5c237","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-12004] Preserve the RDD partitioner through RDD checkpointing\n\nThe solution is the save the RDD partitioner in a separate file in the RDD checkpoint directory. That is, `<checkpoint dir>/_partitioner`.  In most cases, whether the RDD partitioner was recovered or not, does not affect the correctness, only reduces performance. So this solution makes a best-effort attempt to save and recover the partitioner. If either fails, the checkpointing is not affected. This makes this patch safe and backward compatible.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #9983 from tdas/SPARK-12004.\n\n(cherry picked from commit 60b541ee1b97c9e5e84aa2af2ce856f316ad22b3)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/81db8d086bbfe72caa0c45a395ebcaed80b5c237"},{"sha":"21909b8ac0068658cc833f324c0f1f418c200d61","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"Revert \"[SPARK-12060][CORE] Avoid memory copy in JavaSerializerInstance.serialize\"\n\nThis reverts commit 9b99b2b46c452ba396e922db5fc7eec02c45b158.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/21909b8ac0068658cc833f324c0f1f418c200d61"},{"sha":"5647774b07593514f4ed4c29a038cfb1b69c9ba1","author":{"email":"7853928d03425cf3be35be2fa0c5b99641435d26@gmail.com","name":"Xusen Yin"},"message":"[SPARK-11961][DOC] Add docs of ChiSqSelector\n\nhttps://issues.apache.org/jira/browse/SPARK-11961\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\n\nCloses #9965 from yinxusen/SPARK-11961.\n\n(cherry picked from commit e76431f886ae8061545b3216e8e2fb38c4db1f43)\nSigned-off-by: Joseph K. Bradley <joseph@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/5647774b07593514f4ed4c29a038cfb1b69c9ba1"},{"sha":"012de2ce5de01bc57197fa26334fc175c8f20233","author":{"email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com","name":"jerryshao"},"message":"[SPARK-12002][STREAMING][PYSPARK] Fix python direct stream checkpoint recovery issue\n\nFixed a minor race condition in #10017\n\nCloses #10017\n\nAuthor: jerryshao <sshao@hortonworks.com>\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #10074 from zsxwing/review-pr10017.\n\n(cherry picked from commit f292018f8e57779debc04998456ec875f628133b)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/012de2ce5de01bc57197fa26334fc175c8f20233"},{"sha":"d77bf0bd922835b6a63bb1eeedf91e2a92d92ca9","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-12075][SQL] Speed up HiveComparisionTest by avoiding / speeding up TestHive.reset()\n\nWhen profiling HiveCompatibilitySuite, I noticed that most of the time seems to be spent in expensive `TestHive.reset()` calls. This patch speeds up suites based on HiveComparisionTest, such as HiveCompatibilitySuite, with the following changes:\n\n- Avoid `TestHive.reset()` whenever possible:\n  - Use a simple set of heuristics to guess whether we need to call `reset()` in between tests.\n  - As a safety-net, automatically re-run failed tests by calling `reset()` before the re-attempt.\n- Speed up the expensive parts of `TestHive.reset()`: loading the `src` and `srcpart` tables took roughly 600ms per test, so we now avoid this by using a simple heuristic which only loads those tables by tests that reference them. This is based on simple string matching over the test queries which errs on the side of loading in more situations than might be strictly necessary.\n\nAfter these changes, HiveCompatibilitySuite seems to run in about 10 minutes.\n\nThis PR is a revival of #6663, an earlier experimental PR from June, where I played around with several possible speedups for this suite.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10055 from JoshRosen/speculative-testhive-reset.\n\n(cherry picked from commit ef6790fdc3b70b9d6184ec2b3d926e4b0e4b15f6)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/d77bf0bd922835b6a63bb1eeedf91e2a92d92ca9"},{"sha":"f1122dd2bdc4c522a902b37bd34b46f785c21ecf","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@databricks.com","name":"Nong Li"},"message":"[SPARK-11328][SQL] Improve error message when hitting this issue\n\nThe issue is that the output commiter is not idempotent and retry attempts will\nfail because the output file already exists. It is not safe to clean up the file\nas this output committer is by design not retryable. Currently, the job fails\nwith a confusing file exists error. This patch is a stop gap to tell the user\nto look at the top of the error log for the proper message.\n\nThis is difficult to test locally as Spark is hardcoded not to retry. Manually\nverified by upping the retry attempts.\n\nAuthor: Nong Li <nong@databricks.com>\nAuthor: Nong Li <nongli@gmail.com>\n\nCloses #10080 from nongli/spark-11328.\n\n(cherry picked from commit 47a0abc343550c855e679de12983f43e6fcc0171)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/f1122dd2bdc4c522a902b37bd34b46f785c21ecf"},{"sha":"1135430a00dbe6516097dd3bc868ae865e8e644d","author":{"email":"b1011540267ee7fe5a02243ad09c007ca1860809@oc0558782468.ibm.com","name":"Huaxin Gao"},"message":"[SPARK-11788][SQL] surround timestamp/date value with quotes in JDBC data source\n\nWhen query the Timestamp or Date column like the following\nval filtered = jdbcdf.where($\"TIMESTAMP_COLUMN\" >= beg && $\"TIMESTAMP_COLUMN\" < end)\nThe generated SQL query is \"TIMESTAMP_COLUMN >= 2015-01-01 00:00:00.0\"\nIt should have quote around the Timestamp/Date value such as \"TIMESTAMP_COLUMN >= '2015-01-01 00:00:00.0'\"\n\nAuthor: Huaxin Gao <huaxing@oc0558782468.ibm.com>\n\nCloses #9872 from huaxingao/spark-11788.\n\n(cherry picked from commit 5a8b5fdd6ffa58f015cdadf3f2c6df78e0a388ad)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/1135430a00dbe6516097dd3bc868ae865e8e644d"},{"sha":"14eadf921132219f5597d689ac1ffd6e938a939a","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-11352][SQL] Escape */ in the generated comments.\n\nhttps://issues.apache.org/jira/browse/SPARK-11352\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #10072 from yhuai/SPARK-11352.\n\n(cherry picked from commit 5872a9d89fe2720c2bcb1fc7494136947a72581c)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/14eadf921132219f5597d689ac1ffd6e938a939a"},{"sha":"1b3db967e05a628897b7162aa605b2e4650a0d58","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-11596][SQL] In TreeNode's argString, if a TreeNode is not a child of the current TreeNode, we should only return the simpleString.\n\nIn TreeNode's argString, if a TreeNode is not a child of the current TreeNode, we will only return the simpleString.\n\nI tested the [following case provided by Cristian](https://issues.apache.org/jira/browse/SPARK-11596?focusedCommentId=15019241&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15019241).\n```\nval c = (1 to 20).foldLeft[Option[DataFrame]] (None) { (curr, idx) =>\n    println(s\"PROCESSING >>>>>>>>>>> $idx\")\n    val df = sqlContext.sparkContext.parallelize((0 to 10).zipWithIndex).toDF(\"A\", \"B\")\n    val union = curr.map(_.unionAll(df)).getOrElse(df)\n    union.cache()\n    Some(union)\n  }\n\nc.get.explain(true)\n```\n\nWithout the change, `c.get.explain(true)` took 100s. With the change, `c.get.explain(true)` took 26ms.\n\nhttps://issues.apache.org/jira/browse/SPARK-11596\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #10079 from yhuai/SPARK-11596.\n\n(cherry picked from commit e96a70d5ab2e2b43a2df17a550fa9ed2ee0001c4)\nSigned-off-by: Michael Armbrust <michael@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/1b3db967e05a628897b7162aa605b2e4650a0d58"},{"sha":"72da2a21f0940b97757ace5975535e559d627688","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-8414] Ensure context cleaner periodic cleanups\n\nGarbage collection triggers cleanups. If the driver JVM is huge and there is little memory pressure, we may never clean up shuffle files on executors. This is a problem for long-running applications (e.g. streaming).\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #10070 from andrewor14/periodic-gc.\n\n(cherry picked from commit 1ce4adf55b535518c2e63917a827fac1f2df4e8e)\nSigned-off-by: Josh Rosen <joshrosen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/72da2a21f0940b97757ace5975535e559d627688"}]}