{"push_id":608380111,"size":19,"distinct_size":19,"ref":"refs/heads/master","head":"6f80c3e8880340597f161f87e64697bec86cc586","before":"3579003115fa3217cff6aa400729d96b0c7b257b","commits":[{"sha":"6205a255aea0652dddadf953771f5405065b5bec","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-6372] [core] Propagate --conf to child processes.\n\nAnd add unit test.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #5057 from vanzin/SPARK-6372 and squashes the following commits:\n\nb33728b [Marcelo Vanzin] [SPARK-6372] [core] Propagate --conf to child processes.","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/6205a255aea0652dddadf953771f5405065b5bec"},{"sha":"e09c852d6b83b9b112685d113f2792daec8785a3","author":{"email":"ec969b9b5c2f2a49a0d9175cec61f09554863a3d@hortonworks.com","name":"Steve Loughran"},"message":"SPARK-6389 YARN app diagnostics report doesn't report NPEs\n\nTrivial patch to implicitly call `Exception.toString()` over `Exception.getMessage()` —this defaults to including the exception class & any non-null message; some subclasses include more.\n\nNo test.\n\nAuthor: Steve Loughran <stevel@hortonworks.com>\n\nCloses #5070 from steveloughran/stevel/patches/SPARK-6389-NPE-reporting and squashes the following commits:\n\n8239d85 [Steve Loughran] SPARK-6389 cull use of getMessage over toString in the container launcher\n6fbaf6a [Steve Loughran] SPARK-6389 YARN app diagnostics report doesn't report NPEs","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/e09c852d6b83b9b112685d113f2792daec8785a3"},{"sha":"9d112a958ee2facad179344dd367a6d1ccbc9614","author":{"email":"53fdba43521c8d2c2b4ff3a6f81e9f27ad86e7ac@gmail.com","name":"Iulian Dragos"},"message":"[SPARK-6286][minor] Handle missing Mesos case TASK_ERROR.\n\nAuthor: Iulian Dragos <jaguarul@gmail.com>\n\nCloses #5000 from dragos/issue/task-error-case and squashes the following commits:\n\ne063627 [Iulian Dragos] Handle TASK_ERROR in Mesos scheduler backends.\nac17cf0 [Iulian Dragos] Handle missing Mesos case TASK_ERROR.","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/9d112a958ee2facad179344dd367a6d1ccbc9614"},{"sha":"981fbafa2a878e86abeefe1d77cca01fd848f9f6","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-6325] [core,yarn] Do not change target executor count when killing executors.\n\nThe dynamic execution code has two ways to reduce the number of executors: one\nwhere it reduces the total number of executors it wants, by asking for an absolute\nnumber of executors that is lower than the previous one. The second is by\nexplicitly killing idle executors.\n\nYarnAllocator was mixing those up and lowering the target number of executors\nwhen a kill was issued. Instead, trust the frontend knows what it's doing, and kill\nexecutors without messing with other accounting. That means that if the frontend\nkills an executor without lowering the target, it will get a new executor shortly.\n\nThe one situation where both actions (lower the target and kill executor) need to\nhappen together is when user code explicitly calls `SparkContext.killExecutors`.\nIn that case, issue two calls to the backend to achieve the goal.\n\nI also did some minor cleanup in related code:\n- avoid sending a request for executors when target is unchanged, to avoid log\n  spam in the AM\n- avoid printing misleading log messages in the AM when there are no requests\n  to cancel\n- fix a slow memory leak plus misleading error message on the driver caused by\n  failing to completely unregister the executor.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #5018 from vanzin/SPARK-6325 and squashes the following commits:\n\n2e782a3 [Marcelo Vanzin] Avoid redundant logging on the AM side.\na3567cd [Marcelo Vanzin] Add parentheses.\na363926 [Marcelo Vanzin] Update logic.\na158101 [Marcelo Vanzin] [SPARK-6325] [core,yarn] Disallow reducing executor count past running count.","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/981fbafa2a878e86abeefe1d77cca01fd848f9f6"},{"sha":"a95ee242b0a9644c912fc54ed68b4301c9558bc9","author":{"email":"66332c93c294b372ab1bdd5acb9b89a538e2d2f2@gmail.com","name":"Yuhao Yang"},"message":"[SPARK-6374] [MLlib] add get for GeneralizedLinearAlgo\n\nI find it's better to have getter for NumFeatures and addIntercept within GeneralizedLinearAlgorithm during actual usage, otherwise I 'll have to get the value through debug.\n\nAuthor: Yuhao Yang <hhbyyh@gmail.com>\n\nCloses #5058 from hhbyyh/addGetLinear and squashes the following commits:\n\n9dc90e8 [Yuhao Yang] add get for GeneralizedLinearAlgo","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/a95ee242b0a9644c912fc54ed68b4301c9558bc9"},{"sha":"3db13874250ded267d7455898e4048a70a47fdcd","author":{"email":"580258f4dc0f7b7fba6752c3e796a6cb1bdd91fd@gmail.com","name":"Jongyoul Lee"},"message":"SPARK-6085 Part. 2 Increase default value for memory overhead\n\n- fixed a description of spark.mesos.executor.memoryOverhead from 7% to 10%\n- This is a second part of SPARK-6085\n\nAuthor: Jongyoul Lee <jongyoul@gmail.com>\n\nCloses #5065 from jongyoul/SPARK-6085-1 and squashes the following commits:\n\nc5af84c [Jongyoul Lee] SPARK-6085 Part. 2 Increase default value for memory overhead - Changed \"MiB\" to \"MB\"\ndbac1c0 [Jongyoul Lee] SPARK-6085 Part. 2 Increase default value for memory overhead - fixed a description of spark.mesos.executor.memoryOverhead from 7% to 10%","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/3db13874250ded267d7455898e4048a70a47fdcd"},{"sha":"540b2a4eabe0bad2455f5140c4ad6a315e37cc3f","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-6394][Core] cleanup BlockManager companion object and improve the getCacheLocs method in DAGScheduler\n\nThe current implementation include searching a HashMap many times, we can avoid this.\nActually if you look into `BlockManager.blockIdsToBlockManagers`, the core function call is [this](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1258), so we can call `blockManagerMaster.getLocations` directly and avoid building a HashMap.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #5043 from cloud-fan/small and squashes the following commits:\n\ne959d12 [Wenchen Fan] fix style\n203c493 [Wenchen Fan] some cleanup in BlockManager companion object\nd409099 [Wenchen Fan] address rxin's comment\nfaec999 [Wenchen Fan] add regression test\n2fb57aa [Wenchen Fan] imporve the getCacheLocs method","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/540b2a4eabe0bad2455f5140c4ad6a315e37cc3f"},{"sha":"645cf3fcc21987417b2946bdeeeb60af3edf667e","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-6222][Streaming] Dont delete checkpoint data when doing pre-batch-start checkpoint\n\nThis is another alternative approach to https://github.com/apache/spark/pull/4964/\nI think this is a simpler fix that can be backported easily to other branches (1.2 and 1.3).\n\nAll it does it introduce a flag so that the pre-batch-start checkpoint does not call clear checkpoint.\n\nThere is not unit test yet. I will add it when this approach is commented upon. Not sure if this is testable easily.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #5008 from tdas/SPARK-6222 and squashes the following commits:\n\n7315bc2 [Tathagata Das] Removed empty line.\nc438de4 [Tathagata Das] Revert unnecessary change.\n5e98374 [Tathagata Das] Added unit test\n50cb60b [Tathagata Das] Fixed style issue\n295ca5c [Tathagata Das] Fixing SPARK-6222","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/645cf3fcc21987417b2946bdeeeb60af3edf667e"},{"sha":"2c3f83c34bb8d2c1bf13b33633d8c5a8089545d1","author":{"email":"fd726277649d3b2e3ee5d332dca6c89739b68aa5@gmail.com","name":"CodingCat"},"message":"[SPARK-4012] stop SparkContext when the exception is thrown from an infinite loop\n\nhttps://issues.apache.org/jira/browse/SPARK-4012\n\nThis patch is a resubmission for https://github.com/apache/spark/pull/2864\n\nWhat I am proposing in this patch is that ***when the exception is thrown from an infinite loop, we should stop the SparkContext, instead of let JVM throws exception forever***\n\nSo, in the infinite loops where we originally wrapped with a ` logUncaughtExceptions`, I changed to `tryOrStopSparkContext`, so that the Spark component is stopped\n\nEarly stopped JVM process is helpful for HA scheme design, for example,\n\nThe user has a script checking the existence of the pid of the Spark Streaming driver for monitoring the availability; with the code before this patch, the JVM process is still available but not functional when the exceptions are thrown\n\nandrewor14, srowen , mind taking further consideration about the change?\n\nAuthor: CodingCat <zhunansjtu@gmail.com>\n\nCloses #5004 from CodingCat/SPARK-4012-1 and squashes the following commits:\n\n589276a [CodingCat] throw fatal error again\n3c72cd8 [CodingCat] address the comments\n6087864 [CodingCat] revise comments\n6ad3eb0 [CodingCat] stop SparkContext instead of quit the JVM process\n6322959 [CodingCat] exit JVM process when the exception is thrown from an infinite loop","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/2c3f83c34bb8d2c1bf13b33633d8c5a8089545d1"},{"sha":"797f8a000773d848fa52c7fe2eb1b5e5e7f6c55a","author":{"email":"270167d20e82b11e0673d1a46b116e2f4cd206bd@realimpactanalytics.com","name":"Pierre Borckmans"},"message":"[SPARK-6402][DOC] - Remove some refererences to shark in docs and ec2\n\nEC2 script and job scheduling documentation still refered to Shark.\nI removed these references.\n\nI also removed a remaining `SHARK_VERSION` variable from `ec2-variables.sh`.\n\nAuthor: Pierre Borckmans <pierre.borckmans@realimpactanalytics.com>\n\nCloses #5083 from pierre-borckmans/remove_refererences_to_shark_in_docs and squashes the following commits:\n\n4e90ffc [Pierre Borckmans] Removed deprecated SHARK_VERSION\ncaea407 [Pierre Borckmans] Remove shark reference from ec2 script doc\n196c744 [Pierre Borckmans] Removed references to Shark","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/797f8a000773d848fa52c7fe2eb1b5e5e7f6c55a"},{"sha":"3c4e486b9c8d3f256e801db7c176ab650c976135","author":{"email":"2e9bf58ec2f93170404b3a28538fb27400f699db@palantir.com","name":"mcheah"},"message":"[SPARK-5843] [API] Allowing map-side combine to be specified in Java.\n\nSpecifically, when calling JavaPairRDD.combineByKey(), there is a new\nsix-parameter method that exposes the map-side-combine boolean as the\nfifth parameter and the serializer as the sixth parameter.\n\nAuthor: mcheah <mcheah@palantir.com>\n\nCloses #4634 from mccheah/pair-rdd-map-side-combine and squashes the following commits:\n\n5c58319 [mcheah] Fixing compiler errors.\n3ce7deb [mcheah] Addressing style and documentation comments.\n7455c7a [mcheah] Allowing Java combineByKey to specify Serializer as well.\n6ddd729 [mcheah] [SPARK-5843] Allowing map-side combine to be specified in Java.","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/3c4e486b9c8d3f256e801db7c176ab650c976135"},{"sha":"dda4dedca0459fc7c00eb1d9cb07e14af1621e0f","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-6291] [MLLIB] GLM toString & toDebugString\n\nGLM toString prints out intercept, numFeatures.\nFor LogisticRegression and SVM model, toString also prints out numClasses, threshold.\nGLM toDebugString prints out the whole weights, intercept.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #5038 from yanboliang/spark-6291 and squashes the following commits:\n\n2f578b0 [Yanbo Liang] code format\n78b33f2 [Yanbo Liang] fix typos\n1e8a023 [Yanbo Liang] GLM toString & toDebugString","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/dda4dedca0459fc7c00eb1d9cb07e14af1621e0f"},{"sha":"8cb23a1f9a3ed08e57865bcb6cc1cc7902881073","author":{"email":"eaca3ca4b51928dd768a043ece15380fadcb1b95@capitalone.com","name":"Brennon York"},"message":"[SPARK-5313][Project Infra]: Create simple framework for highlighting changes introduced in a PR\n\nBuilt a simple framework with a `dev/tests` directory to house all pull request related tests. I've moved the two original tests (`pr_merge_ability` and `pr_public_classes`) into the new `dev/tests` directory and tested to the best of my ability. At this point I need to test against Jenkins actually running the new `run-tests-jenkins` script to ensure things aren't broken down the path.\n\nAuthor: Brennon York <brennon.york@capitalone.com>\n\nCloses #5072 from brennonyork/SPARK-5313 and squashes the following commits:\n\n8ae990c [Brennon York] added dev/run-tests back, removed echo\n5db4ed4 [Brennon York] removed the git checkout\n1b50050 [Brennon York] adding echos to see what jenkins is seeing\nb823959 [Brennon York] removed run-tests to further test the public_classes pr test\n2b9ce12 [Brennon York] added the dev/run-tests call back in\nffd49c0 [Brennon York] remove -c from bash as that was removing the trailing args\n735d615 [Brennon York] removed the actual dev/run-tests command to further test jenkins\nd579662 [Brennon York] Merge remote-tracking branch 'upstream/master' into SPARK-5313\naa48029 [Brennon York] removed echo lines for testing jenkins\n24cd965 [Brennon York] added test output to check within jenkins to verify\n3a38e73 [Brennon York] removed the temporary read\n9c881ff [Brennon York] updated test suite\n183b7ee [Brennon York] added documentation on how to create tests\n0bc2efe [Brennon York] ensure each test starts on the current pr branch\n1743378 [Brennon York] added tests in test suite\nabd7430 [Brennon York] updated to include test suite","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/8cb23a1f9a3ed08e57865bcb6cc1cc7902881073"},{"sha":"3b5aaa6a5fe0d838a8570c9d500ebca5f63764f8","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[Core][minor] remove unused `visitedStages` in `DAGScheduler.stageDependsOn`\n\nWe define and update `visitedStages` in `DAGScheduler.stageDependsOn`, but never read it. So we can safely remove it.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #5086 from cloud-fan/minor and squashes the following commits:\n\n24663ea [Wenchen Fan] remove un-used variable","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/3b5aaa6a5fe0d838a8570c9d500ebca5f63764f8"},{"sha":"f17d43b033d928dbc46aef8e367aa08902e698ad","author":{"email":"fbfd45890515f7eeeeabf87d7a80d3449c3bce83@gmail.com","name":"Nicholas Chammas"},"message":"[SPARK-6219] [Build] Check that Python code compiles\n\nThis PR expands the Python lint checks so that they check for obvious compilation errors in our Python code.\n\nFor example:\n\n```\n$ ./dev/lint-python\nPython lint checks failed.\nCompiling ./ec2/spark_ec2.py ...\n  File \"./ec2/spark_ec2.py\", line 618\n    return (master_nodes,, slave_nodes)\n                         ^\nSyntaxError: invalid syntax\n\n./ec2/spark_ec2.py:618:25: E231 missing whitespace after ','\n./ec2/spark_ec2.py:1117:101: E501 line too long (102 > 100 characters)\n```\n\nThis PR also bumps up the version of `pep8`. It ignores new types of checks introduced by that version bump while fixing problems missed by the older version of `pep8` we were using.\n\nAuthor: Nicholas Chammas <nicholas.chammas@gmail.com>\n\nCloses #4941 from nchammas/compile-spark-ec2 and squashes the following commits:\n\n75e31d8 [Nicholas Chammas] upgrade pep8 + check compile\nb33651c [Nicholas Chammas] PEP8 line length","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/f17d43b033d928dbc46aef8e367aa08902e698ad"},{"sha":"0745a305fac622a6eeb8aa4a7401205a14252939","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Tighten up field/method visibility in Executor and made some code more clear to read.\n\nI was reading Executor just now and found that some latest changes introduced some weird code path with too much monadic chaining and unnecessary fields. I cleaned it up a bit, and also tightened up the visibility of various fields/methods. Also added some inline documentation to help understand this code better.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #4850 from rxin/executor and squashes the following commits:\n\n866fc60 [Reynold Xin] Code review feedback.\n020efbb [Reynold Xin] Tighten up field/method visibility in Executor and made some code more clear to read.","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/0745a305fac622a6eeb8aa4a7401205a14252939"},{"sha":"116c553fd6f6d2adcbbf000cd80b5c46d4516e87","author":{"email":"580258f4dc0f7b7fba6752c3e796a6cb1bdd91fd@gmail.com","name":"Jongyoul Lee"},"message":"[SPARK-6286][Mesos][minor] Handle missing Mesos case TASK_ERROR\n\n- Made TaskState.isFailed for handling TASK_LOST and TASK_ERROR and synchronizing CoarseMesosSchedulerBackend and MesosSchedulerBackend\n- This is related #5000\n\nAuthor: Jongyoul Lee <jongyoul@gmail.com>\n\nCloses #5088 from jongyoul/SPARK-6286-1 and squashes the following commits:\n\n4f2362f [Jongyoul Lee] [SPARK-6286][Mesos][minor] Handle missing Mesos case TASK_ERROR - Fixed scalastyle\nac4336a [Jongyoul Lee] [SPARK-6286][Mesos][minor] Handle missing Mesos case TASK_ERROR - Made TaskState.isFailed for handling TASK_LOST and TASK_ERROR and synchronizing CoarseMesosSchedulerBackend and MesosSchedulerBackend","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/116c553fd6f6d2adcbbf000cd80b5c46d4516e87"},{"sha":"d08e3eb3dc455970b685a7b8b7e00c537c89a8e4","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"SPARK-5134 [BUILD] Bump default Hadoop version to 2+\n\nBump default Hadoop version to 2.2.0. (This is already the dependency version reported by published Maven artifacts.) See JIRA for further discussion.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #5027 from srowen/SPARK-5134 and squashes the following commits:\n\nacbee14 [Sean Owen] Bump default Hadoop version to 2.2.0. (This is already the dependency version reported by published Maven artifacts.)","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/d08e3eb3dc455970b685a7b8b7e00c537c89a8e4"},{"sha":"6f80c3e8880340597f161f87e64697bec86cc586","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":" SPARK-6338 [CORE] Use standard temp dir mechanisms in tests to avoid orphaned temp files\n\nUse `Utils.createTempDir()` to replace other temp file mechanisms used in some tests, to further ensure they are cleaned up, and simplify\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #5029 from srowen/SPARK-6338 and squashes the following commits:\n\n27b740a [Sean Owen] Fix hive-thriftserver tests that don't expect an existing dir\n4a212fa [Sean Owen] Standardize a bit more temp dir management\n9004081 [Sean Owen] Revert some added recursive-delete calls\n57609e4 [Sean Owen] Use Utils.createTempDir() to replace other temp file mechanisms used in some tests, to further ensure they are cleaned up, and simplify","distinct":true,"url":"https://api.github.com/repos/rakeshchalasani/spark/commits/6f80c3e8880340597f161f87e64697bec86cc586"}]}