{"push_id":868016029,"size":19,"distinct_size":19,"ref":"refs/heads/branch-1.6","head":"e12ecfa360c54add488096493ce0dba15cbf3f8d","before":"a0f9cd77a73ae911d382ac8ddd52614800663704","commits":[{"sha":"c37ed52ecd38a400f99cc82f07440e455b772db2","author":{"email":"3aa38c00921c7acfa71de69042a28a39f706d59b@us.ibm.com","name":"xin Wu"},"message":"[SPARK-11522][SQL] input_file_name() returns \"\" for external tables\n\nWhen computing partition for non-parquet relation, `HadoopRDD.compute` is used. but it does not set the thread local variable `inputFileName` in `NewSqlHadoopRDD`, like `NewSqlHadoopRDD.compute` does.. Yet, when getting the `inputFileName`, `NewSqlHadoopRDD.inputFileName` is exptected, which is empty now.\nAdding the setting inputFileName in HadoopRDD.compute resolves this issue.\n\nAuthor: xin Wu <xinwu@us.ibm.com>\n\nCloses #9542 from xwu0226/SPARK-11522.\n\n(cherry picked from commit 0e79604aed116bdcb40e03553a2d103b5b1cdbae)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/c37ed52ecd38a400f99cc82f07440e455b772db2"},{"sha":"949c9b7c660c37bbe543adb260380afb1258089e","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11752] [SQL] fix timezone problem for DateTimeUtils.getSeconds\n\ncode snippet to reproduce it:\n```\nTimeZone.setDefault(TimeZone.getTimeZone(\"Asia/Shanghai\"))\nval t = Timestamp.valueOf(\"1900-06-11 12:14:50.789\")\nval us = fromJavaTimestamp(t)\nassert(getSeconds(us) === t.getSeconds)\n```\n\nit will be good to add a regression test for it, but the reproducing code need to change the default timezone, and even we change it back, the `lazy val defaultTimeZone` in `DataTimeUtils` is fixed.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9728 from cloud-fan/seconds.\n\n(cherry picked from commit 06f1fdba6d1425afddfc1d45a20dbe9bede15e7a)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/949c9b7c660c37bbe543adb260380afb1258089e"},{"sha":"1887fa228f455577e41b754262d5b0addfe50c22","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-11743] [SQL] Add UserDefinedType support to RowEncoder\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-11743\n\nRowEncoder doesn't support UserDefinedType now. We should add the support for it.\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #9712 from viirya/rowencoder-udt.\n\n(cherry picked from commit b0c3fd34e4cfa3f0472d83e71ffe774430cfdc87)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/1887fa228f455577e41b754262d5b0addfe50c22"},{"sha":"f14fb291d822720d8b578db0bdb656fb6c9ce590","author":{"email":"2680965ad5aae29275c8fb10150cb10f5554e78b@gmail.com","name":"hyukjinkwon"},"message":"[SPARK-11044][SQL] Parquet writer version fixed as version1\n\nhttps://issues.apache.org/jira/browse/SPARK-11044\n\nSpark writes a parquet file only with writer version1 ignoring the writer version given by user.\n\nSo, in this PR, it keeps the writer version if given or sets version1 as default.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\nAuthor: HyukjinKwon <gurwls223@gmail.com>\n\nCloses #9060 from HyukjinKwon/SPARK-11044.\n\n(cherry picked from commit 7f8eb3bf6ed64eefc5472f5c5fb02e2db1e3f618)\nSigned-off-by: Cheng Lian <lian@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/f14fb291d822720d8b578db0bdb656fb6c9ce590"},{"sha":"38673d7e6358622f240d7331b061cadb96f8409f","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":"[SPARK-11731][STREAMING] Enable batching on Driver WriteAheadLog by default\n\nUsing batching on the driver for the WriteAheadLog should be an improvement for all environments and use cases. Users will be able to scale to much higher number of receivers with the BatchedWriteAheadLog. Therefore we should turn it on by default, and QA it in the QA period.\n\nI've also added some tests to make sure the default configurations are correct regarding recent additions:\n - batching on by default\n - closeFileAfterWrite off by default\n - parallelRecovery off by default\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #9695 from brkyvz/enable-batch-wal.\n\n(cherry picked from commit de5e531d337075fd849437e88846873bca8685e6)\nSigned-off-by: Tathagata Das <tathagata.das1565@gmail.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/38673d7e6358622f240d7331b061cadb96f8409f"},{"sha":"c83177d3045afccf07465c43caac74d980673f42","author":{"email":"6f2531f6e4f53ed242dd8de1f010b03aebb8d02b@us.ibm.com","name":"Daniel Jalova"},"message":"[SPARK-6328][PYTHON] Python API for StreamingListener\n\nAuthor: Daniel Jalova <djalova@us.ibm.com>\n\nCloses #9186 from djalova/SPARK-6328.\n\n(cherry picked from commit ace0db47141ffd457c2091751038fc291f6d5a8b)\nSigned-off-by: Tathagata Das <tathagata.das1565@gmail.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/c83177d3045afccf07465c43caac74d980673f42"},{"sha":"38fe092ff3a1e86088a22adc934a4b4e269b0a47","author":{"email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com","name":"jerryshao"},"message":"[SPARK-11718][YARN][CORE] Fix explicitly killed executor dies silently issue\n\nCurrently if dynamic allocation is enabled, explicitly killing executor will not get response, so the executor metadata is wrong in driver side. Which will make dynamic allocation on Yarn fail to work.\n\nThe problem is  `disableExecutor` returns false for pending killing executors when `onDisconnect` is detected, so no further implementation is done.\n\nOne solution is to bypass these explicitly killed executors to use `super.onDisconnect` to remove executor. This is simple.\n\nAnother solution is still querying the loss reason for these explicitly kill executors. Since executor may get killed and informed in the same AM-RM communication, so current way of adding pending loss reason request is not worked (container complete is already processed), here we should store this loss reason for later query.\n\nHere this PR chooses solution 2.\n\nPlease help to review. vanzin I think this part is changed by you previously, would you please help to review? Thanks a lot.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #9684 from jerryshao/SPARK-11718.\n\n(cherry picked from commit 24477d2705bcf2a851acc241deb8376c5450dc73)\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/38fe092ff3a1e86088a22adc934a4b4e269b0a47"},{"sha":"fbe65c5924d2f5f4789bf54a1da0a7b6bbf1eb42","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11754][SQL] consolidate `ExpressionEncoder.tuple` and `Encoders.tuple`\n\nThese 2 are very similar, we can consolidate them into one.\n\nAlso add tests for it and fix a bug.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9729 from cloud-fan/tuple.\n\n(cherry picked from commit b1a9662623951079e80bd7498e064c4cae4977e9)\nSigned-off-by: Michael Armbrust <michael@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/fbe65c5924d2f5f4789bf54a1da0a7b6bbf1eb42"},{"sha":"90d71bff0c583830aa3fd96b1dd3607f0cb0cbee","author":{"email":"2a23f60d9b3f5d0192f4f989e8da0b6c87155a05@us.ibm.com","name":"Zee Chen"},"message":"[SPARK-11390][SQL] Query plan with/without filterPushdown indistinguishable\n\n…ishable\n\nPropagate pushed filters to PhyicalRDD in DataSourceStrategy.apply\n\nAuthor: Zee Chen <zeechen@us.ibm.com>\n\nCloses #9679 from zeocio/spark-11390.\n\n(cherry picked from commit 985b38dd2fa5d8f1e23f1c420ce6262e7e3ed181)\nSigned-off-by: Michael Armbrust <michael@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/90d71bff0c583830aa3fd96b1dd3607f0cb0cbee"},{"sha":"64439f7d61edfbaa5513ddcf3f9dc86fa905aef2","author":{"email":"399a71a219a2bbf15a9ad52c797d50eb921e7e28@gmail.com","name":"Davies Liu"},"message":"Revert \"[SPARK-11271][SPARK-11016][CORE] Use Spark BitSet instead of RoaringBitmap to reduce memory usage\"\n\nThis reverts commit e209fa271ae57dc8849f8b1241bf1ea7d6d3d62c.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/64439f7d61edfbaa5513ddcf3f9dc86fa905aef2"},{"sha":"3bd72eafc5b7204d49fc1a3d587337fae88d8ae8","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-11742][STREAMING] Add the failure info to the batch lists\n\n<img width=\"1365\" alt=\"screen shot 2015-11-13 at 9 57 43 pm\" src=\"https://cloud.githubusercontent.com/assets/1000778/11162322/9b88e204-8a51-11e5-8c57-a44889cab713.png\">\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #9711 from zsxwing/failure-info.\n\n(cherry picked from commit bcea0bfda66a30ee86790b048de5cb47b4d0b32f)\nSigned-off-by: Tathagata Das <tathagata.das1565@gmail.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/3bd72eafc5b7204d49fc1a3d587337fae88d8ae8"},{"sha":"6c8e0c0ff4fd3acbe0dbe7119920ee8b74d191dd","author":{"email":"84aac467eb42e080528ac0b2042a1322416f9eb0@allegrogroup.com","name":"Bartlomiej Alberski"},"message":"[SPARK-11553][SQL] Primitive Row accessors should not convert null to default value\n\nInvocation of getters for type extending AnyVal returns default value (if field value is null) instead of throwing NPE. Please check comments for SPARK-11553 issue for more details.\n\nAuthor: Bartlomiej Alberski <bartlomiej.alberski@allegrogroup.com>\n\nCloses #9642 from alberskib/bugfix/SPARK-11553.\n\n(cherry picked from commit 31296628ac7cd7be71e0edca335dc8604f62bb47)\nSigned-off-by: Michael Armbrust <michael@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/6c8e0c0ff4fd3acbe0dbe7119920ee8b74d191dd"},{"sha":"e042780cd95f5bf3988dd73bc4431c3b012c171c","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-8658][SQL] AttributeReference's equals method compares all the members\n\nThis fix is to change the equals method to check all of the specified fields for equality of AttributeReference.\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #9216 from gatorsmile/namedExpressEqual.\n\n(cherry picked from commit 75ee12f09c2645c1ad682764d512965f641eb5c2)\nSigned-off-by: Michael Armbrust <michael@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/e042780cd95f5bf3988dd73bc4431c3b012c171c"},{"sha":"4f8c7e18f3103ee1fcf5a79c1d39cf5a81e78c87","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11625][SQL] add java test for typed aggregate\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9591 from cloud-fan/agg-test.\n\n(cherry picked from commit fd14936be7beff543dbbcf270f2f9749f7a803c4)\nSigned-off-by: Michael Armbrust <michael@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/4f8c7e18f3103ee1fcf5a79c1d39cf5a81e78c87"},{"sha":"bb044ec2278e5623e8fd3ae1f3172b5047b83439","author":{"email":"ad63b47dfac1951bde2b661ba9fbbe123f686bee@cs.berkeley.edu","name":"Shivaram Venkataraman"},"message":"[SPARKR][HOTFIX] Disable flaky SparkR package build test\n\nSee https://github.com/apache/spark/pull/9390#issuecomment-157160063 and https://gist.github.com/shivaram/3a2fecce60768a603dac for more information\n\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\n\nCloses #9744 from shivaram/sparkr-package-test-disable.\n\n(cherry picked from commit ea6f53e48a911b49dc175ccaac8c80e0a1d97a09)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/bb044ec2278e5623e8fd3ae1f3172b5047b83439"},{"sha":"e4abfe9324e3fdebace8b33cc6616f382ad7ac45","author":{"email":"7d45abe39fd242ee2f6575da28f13d66568418b3@oss.nttdata.co.jp","name":"Kousuke Saruta"},"message":"[SPARK-11480][CORE][WEBUI] Wrong callsite is displayed when using AsyncRDDActions#takeAsync\n\nWhen we call AsyncRDDActions#takeAsync, actually another DAGScheduler#runJob is called from another thread so we cannot get proper callsite infomation.\n\nFollowing screenshots are before this patch applied and after.\n\nBefore:\n<img width=\"1268\" alt=\"2015-11-04 1 26 40\" src=\"https://cloud.githubusercontent.com/assets/4736016/10914069/0ffc1306-8294-11e5-8e89-c4fadf58dd12.png\">\n<img width=\"1258\" alt=\"2015-11-04 1 26 52\" src=\"https://cloud.githubusercontent.com/assets/4736016/10914070/0ffe84ce-8294-11e5-8b2a-69d36276bedb.png\">\n\nAfter:\n<img width=\"1268\" alt=\"2015-11-04 0 48 07\" src=\"https://cloud.githubusercontent.com/assets/4736016/10914080/1d8cfb7a-8294-11e5-9e09-ede25c2563e8.png\">\n<img width=\"1269\" alt=\"2015-11-04 0 48 26\" src=\"https://cloud.githubusercontent.com/assets/4736016/10914081/1d934e3a-8294-11e5-8b5e-e3dc37aaced3.png\">\n\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #9437 from sarutak/SPARK-11480.\n\n(cherry picked from commit 30f3cfda1cce8760f15c0a48a8c47f09a5167fca)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/e4abfe9324e3fdebace8b33cc6616f382ad7ac45"},{"sha":"32a69e4c108de8faae8244df7665de717b89e747","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-11710] Document new memory management model\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #9676 from andrewor14/memory-management-docs.\n\n(cherry picked from commit 33a0ec93771ef5c3b388165b07cfab9014918d3b)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/32a69e4c108de8faae8244df7665de717b89e747"},{"sha":"505eceef303e3291253b35164fbec7e4390e8252","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-11612][ML] Pipeline and PipelineModel persistence\n\nPipeline and PipelineModel extend Readable and Writable.  Persistence succeeds only when all stages are Writable.\n\nNote: This PR reinstates tests for other read/write functionality.  It should probably not get merged until [https://issues.apache.org/jira/browse/SPARK-11672] gets fixed.\n\nCC: mengxr\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #9674 from jkbradley/pipeline-io.\n\n(cherry picked from commit 1c5475f1401d2233f4c61f213d1e2c2ee9673067)\nSigned-off-by: Joseph K. Bradley <joseph@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/505eceef303e3291253b35164fbec7e4390e8252"},{"sha":"e12ecfa360c54add488096493ce0dba15cbf3f8d","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-11617][NETWORK] Fix leak in TransportFrameDecoder.\n\nThe code was using the wrong API to add data to the internal composite\nbuffer, causing buffers to leak in certain situations. Use the right\nAPI and enhance the tests to catch memory leaks.\n\nAlso, avoid reusing the composite buffers when downstream handlers keep\nreferences to them; this seems to cause a few different issues even though\nthe ref counting code seems to be correct, so instead pay the cost of copying\na few bytes when that situation happens.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #9619 from vanzin/SPARK-11617.\n\n(cherry picked from commit 540bf58f18328c68107d6c616ffd70f3a4640054)\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/e12ecfa360c54add488096493ce0dba15cbf3f8d"}]}