{"push_id":890887035,"size":26,"distinct_size":26,"ref":"refs/heads/master","head":"95296d9b1ad1d9e9396d7dfd0015ef27ce1cf341","before":"4375eb3f48fc7ae90caf6c21a0d3ab0b66bf4efa","commits":[{"sha":"128c29035b4e7383cc3a9a6c7a9ab6136205ac6c","author":{"email":"86548942b2ff162d27014225aec68c3b73c8489a@surfsara.nl","name":"Jeroen Schot"},"message":"[SPARK-3580][CORE] Add Consistent Method To Get Number of RDD Partitions Across Different Languages\n\nI have tried to address all the comments in pull request https://github.com/apache/spark/pull/2447.\n\nNote that the second commit (using the new method in all internal code of all components) is quite intrusive and could be omitted.\n\nAuthor: Jeroen Schot <jeroen.schot@surfsara.nl>\n\nCloses #9767 from schot/master.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/128c29035b4e7383cc3a9a6c7a9ab6136205ac6c"},{"sha":"a1542ce2f33ad365ff437d2d3014b9de2f6670e5","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-12094][SQL] Prettier tree string for TreeNode\n\nWhen examining plans of complex queries with multiple joins, a pain point of mine is that, it's hard to immediately see the sibling node of a specific query plan node. This PR adds tree lines for the tree string of a `TreeNode`, so that the result can be visually more intuitive.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #10099 from liancheng/prettier-tree-string.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/a1542ce2f33ad365ff437d2d3014b9de2f6670e5"},{"sha":"452690ba1cc3c667bdd9f3022c43c9a10267880b","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-12001] Allow partially-stopped StreamingContext to be completely stopped\n\nIf `StreamingContext.stop()` is interrupted midway through the call, the context will be marked as stopped but certain state will have not been cleaned up. Because `state = STOPPED` will be set, subsequent `stop()` calls will be unable to finish stopping the context, preventing any new StreamingContexts from being created.\n\nThis patch addresses this issue by only marking the context as `STOPPED` once the `stop()` has successfully completed which allows `stop()` to be called a second time in order to finish stopping the context in case the original `stop()` call was interrupted.\n\nI discovered this issue by examining logs from a failed Jenkins run in which this race condition occurred in `FailureSuite`, leaking an unstoppable context and causing all subsequent tests to fail.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9982 from JoshRosen/SPARK-12001.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/452690ba1cc3c667bdd9f3022c43c9a10267880b"},{"sha":"de07d06abecf3516c95d099b6c01a86e0c8cfd8c","author":{"email":"bc4a65f227ddcd81bb538c67e810a3e11696e8bf@gmail.com","name":"Yu ISHIKAWA"},"message":"[SPARK-10266][DOCUMENTATION, ML] Fixed @Since annotation for ml.tunning\n\ncc mengxr noel-smith\n\nI worked on this issues based on https://github.com/apache/spark/pull/8729.\nehsanmok  thank you for your contricution!\n\nAuthor: Yu ISHIKAWA <yuu.ishikawa@gmail.com>\nAuthor: Ehsan M.Kermani <ehsanmo1367@gmail.com>\n\nCloses #9338 from yu-iskw/JIRA-10266.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/de07d06abecf3516c95d099b6c01a86e0c8cfd8c"},{"sha":"d0d7ec533062151269b300ed455cf150a69098c0","author":{"email":"88ec0e7df70fbb4dfef58da07dfeb1a4a708d199@gmail.com","name":"Yadong Qi"},"message":"[SPARK-12093][SQL] Fix the error of comment in DDLParser\n\nAuthor: Yadong Qi <qiyadong2010@gmail.com>\n\nCloses #10096 from watermen/patch-1.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/d0d7ec533062151269b300ed455cf150a69098c0"},{"sha":"9bb695b7a82d837e2c7a724514ea6b203efb5364","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-12000] do not specify arg types when reference a method in ScalaDoc\n\nThis fixes SPARK-12000, verified on my local with JDK 7. It seems that `scaladoc` try to match method names and messed up with annotations.\n\ncc: JoshRosen jkbradley\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #10114 from mengxr/SPARK-12000.2.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/9bb695b7a82d837e2c7a724514ea6b203efb5364"},{"sha":"ae402533738be06ac802914ed3e48f0d5fa54cbe","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-12082][FLAKY-TEST] Increase timeouts in NettyBlockTransferSecuritySuite\n\nWe should try increasing a timeout in NettyBlockTransferSecuritySuite in order to reduce that suite's flakiness in Jenkins.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10113 from JoshRosen/SPARK-12082.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/ae402533738be06ac802914ed3e48f0d5fa54cbe"},{"sha":"ec2b6c26c9b6bd59d29b5d7af2742aca7e6e0b07","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-12109][SQL] Expressions's simpleString should delegate to its toString.\n\nhttps://issues.apache.org/jira/browse/SPARK-12109\n\nThe change of https://issues.apache.org/jira/browse/SPARK-11596 exposed the problem.\nIn the sql plan viz, the filter shows\n\n![image](https://cloud.githubusercontent.com/assets/2072857/11547075/1a285230-9906-11e5-8481-2bb451e35ef1.png)\n\nAfter changes in this PR, the viz is back to normal.\n![image](https://cloud.githubusercontent.com/assets/2072857/11547080/2bc570f4-9906-11e5-8897-3b3bff173276.png)\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #10111 from yhuai/SPARK-12109.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/ec2b6c26c9b6bd59d29b5d7af2742aca7e6e0b07"},{"sha":"5349851f368a1b5dab8a99c0d51c9638ce7aec56","author":{"email":"b1011540267ee7fe5a02243ad09c007ca1860809@oc0558782468.ibm.com","name":"Huaxin Gao"},"message":"[SPARK-12088][SQL] check connection.isClosed before calling connection…\n\nIn Java Spec java.sql.Connection, it has\nboolean getAutoCommit() throws SQLException\nThrows:\nSQLException - if a database access error occurs or this method is called on a closed connection\n\nSo if conn.getAutoCommit is called on a closed connection, a SQLException will be thrown. Even though the code catch the SQLException and program can continue, I think we should check conn.isClosed before calling conn.getAutoCommit to avoid the unnecessary SQLException.\n\nAuthor: Huaxin Gao <huaxing@oc0558782468.ibm.com>\n\nCloses #10095 from huaxingao/spark-12088.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/5349851f368a1b5dab8a99c0d51c9638ce7aec56"},{"sha":"7470d9edbb0a45e714c96b5d55eff30724c0653a","author":{"email":"ced9a24e2e494bb60341a9f26a61a78e7a11abad@apache.org","name":"Jeff Zhang"},"message":"[DOCUMENTATION][MLLIB] typo in mllib doc\n\n\\cc mengxr\n\nAuthor: Jeff Zhang <zjffdu@apache.org>\n\nCloses #10093 from zjffdu/mllib_typo.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/7470d9edbb0a45e714c96b5d55eff30724c0653a"},{"sha":"95b3cf125b905b4ef8705c46f2ef255377b0a9dc","author":{"email":"323adf4c05fd4cdb2f7e46a9360e9eb3eecb2833@kaiyuandao.com","name":"microwishing"},"message":"[DOCUMENTATION][KAFKA] fix typo in kafka/OffsetRange.scala\n\nthis is to fix some typo in external/kafka/src/main/scala/org/apache/spark/streaming/kafka/OffsetRange.scala\n\nAuthor: microwishing <wei.zhu@kaiyuandao.com>\n\nCloses #10121 from microwishing/master.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/95b3cf125b905b4ef8705c46f2ef255377b0a9dc"},{"sha":"43c575cb1766b32c74db17216194a8a74119b759","author":{"email":"fd68b5404b021a07e5ec5abdeb1b0ffe8f04e45b@hotmail.com","name":"felixcheung"},"message":"[SPARK-12116][SPARKR][DOCS] document how to workaround function name conflicts with dplyr\n\nshivaram\n\nAuthor: felixcheung <felixcheung_m@hotmail.com>\n\nCloses #10119 from felixcheung/rdocdplyrmasked.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/43c575cb1766b32c74db17216194a8a74119b759"},{"sha":"8fa3e474a8ba180188361c0ad7e2704c3e2258d3","author":{"email":"ec969b9b5c2f2a49a0d9175cec61f09554863a3d@hortonworks.com","name":"Steve Loughran"},"message":"[SPARK-11314][YARN] add service API and test service for Yarn Cluster schedulers\n\nThis is purely the yarn/src/main and yarn/src/test bits of the YARN ATS integration: the extension model to load and run implementations of `SchedulerExtensionService` in the yarn cluster scheduler process —and to stop them afterwards.\n\nThere's duplication between the two schedulers, yarn-client and yarn-cluster, at least in terms of setting everything up, because the common superclass, `YarnSchedulerBackend` is in spark-core, and the extension services need the YARN app/attempt IDs.\n\nIf you look at how the the extension services are loaded, the case class `SchedulerExtensionServiceBinding` is used to pass in config info -currently just the spark context and the yarn IDs, of which one, the attemptID, will be null when running client-side. I'm passing in a case class to ensure that it would be possible in future to add extra arguments to the binding class, yet, as the method signature will not have changed, still be able to load existing services.\n\nThere's no functional extension service here, just one for testing. The real tests come in the bigger pull requests. At the same time, there's no restriction of this extension service purely to the ATS history publisher. Anything else that wants to listen to the spark context and publish events could use this, and I'd also consider writing one for the YARN-913 registry service, so that the URLs of the web UI would be locatable through that (low priority; would make more sense if integrated with a REST client).\n\nThere's no minicluster test. Given the test execution overhead of setting up minicluster tests, it'd  probably be better to add an extension service into one of the existing tests.\n\nAuthor: Steve Loughran <stevel@hortonworks.com>\n\nCloses #9182 from steveloughran/stevel/feature/SPARK-1537-service.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/8fa3e474a8ba180188361c0ad7e2704c3e2258d3"},{"sha":"7bc9e1db2c47387ee693bcbeb4a8a2cbe11909cf","author":{"email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com","name":"jerryshao"},"message":"[SPARK-12059][CORE] Avoid assertion error when unexpected state transition met in Master\n\nDowngrade to warning log for unexpected state transition.\n\nandrewor14 please review, thanks a lot.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #10091 from jerryshao/SPARK-12059.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/7bc9e1db2c47387ee693bcbeb4a8a2cbe11909cf"},{"sha":"649be4fa4532dcd3001df8345f9f7e970a3fbc65","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-12101][CORE] Fix thread pools that cannot cache tasks in Worker and AppClient\n\n`SynchronousQueue` cannot cache any task. This issue is similar to #9978. It's an easy fix. Just use the fixed `ThreadUtils.newDaemonCachedThreadPool`.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #10108 from zsxwing/fix-threadpool.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/649be4fa4532dcd3001df8345f9f7e970a3fbc65"},{"sha":"688e521c2833a00069272a6749153d721a0996f6","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-12108] Make event logs smaller\n\n**Problem.** Event logs in 1.6 were much bigger than 1.5. I ran page rank and the event log size in 1.6 was almost 5x that in 1.5. I did a bisect to find that the RDD callsite added in #9398 is largely responsible for this.\n\n**Solution.** This patch removes the long form of the callsite (which is not used!) from the event log. This reduces the size of the event log significantly.\n\n*Note on compatibility*: if this patch is to be merged into 1.6.0, then it won't break any compatibility. Otherwise, if it is merged into 1.6.1, then we might need to add more backward compatibility handling logic (currently does not exist yet).\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #10115 from andrewor14/smaller-event-logs.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/688e521c2833a00069272a6749153d721a0996f6"},{"sha":"d576e76bbaa818480d31d2b8fbbe4b15718307d9","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[MINOR][ML] Use coefficients replace weights\n\nUse ```coefficients``` replace ```weights```, I wish they are the last two.\nmengxr\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #10065 from yanboliang/coefficients.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/d576e76bbaa818480d31d2b8fbbe4b15718307d9"},{"sha":"ad7cea6f776a39801d6bb5bb829d1800b175b2ab","author":{"email":"fbfd45890515f7eeeeabf87d7a80d3449c3bce83@gmail.com","name":"Nicholas Chammas"},"message":"[SPARK-12107][EC2] Update spark-ec2 versions\n\nI haven't created a JIRA. If we absolutely need one I'll do it, but I'm fine with not getting mentioned in the release notes if that's the only purpose it'll serve.\n\ncc marmbrus - We should include this in 1.6-RC2 if there is one. I can open a second PR against branch-1.6 if necessary.\n\nAuthor: Nicholas Chammas <nicholas.chammas@gmail.com>\n\nCloses #10109 from nchammas/spark-ec2-versions.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/ad7cea6f776a39801d6bb5bb829d1800b175b2ab"},{"sha":"a02d47277379e1e82d0ee41b2205434f9ffbc3e5","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[FLAKY-TEST-FIX][STREAMING][TEST] Make sure StreamingContexts are shutdown after test\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #10124 from tdas/InputStreamSuite-flaky-test.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/a02d47277379e1e82d0ee41b2205434f9ffbc3e5"},{"sha":"2213441e5e0fba01e05826257604aa427cdf2598","author":{"email":"fd68b5404b021a07e5ec5abdeb1b0ffe8f04e45b@hotmail.com","name":"felixcheung"},"message":"[SPARK-12019][SPARKR] Support character vector for sparkR.init(), check param and fix doc\n\nand add tests.\nSpark submit expects comma-separated list\n\nAuthor: felixcheung <felixcheung_m@hotmail.com>\n\nCloses #10034 from felixcheung/sparkrinitdoc.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/2213441e5e0fba01e05826257604aa427cdf2598"}]}