{"push_id":560750745,"size":242,"distinct_size":242,"ref":"refs/heads/master","head":"6f21dce5f4619e1a5d07028e2a74dc36be0849b9","before":"cdccc263b20c1bb27b864411c82cfad7daca1f47","commits":[{"sha":"342612b65f3d77c660383a332f0346872f076647","author":{"email":"0e98f9ac65f56eff35c0200af71769b196f061ba@sigmoidanalytics.com","name":"sigmoidanalytics"},"message":"[SPARK-5058] Updated broken links\n\nUpdated the broken link pointing to the KafkaWordCount example to the correct one.\n\nAuthor: sigmoidanalytics <mayur@sigmoidanalytics.com>\n\nCloses #3877 from sigmoidanalytics/patch-1 and squashes the following commits:\n\n3e19b31 [sigmoidanalytics] Updated broken links","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/342612b65f3d77c660383a332f0346872f076647"},{"sha":"b96008d5529bac5fd57b76554fd01760139cffff","author":{"email":"eaca3ca4b51928dd768a043ece15380fadcb1b95@capitalone.com","name":"Brennon York"},"message":"[SPARK-794][Core] Remove sleep() in ClusterScheduler.stop\n\nRemoved `sleep()` from the `stop()` method of the `TaskSchedulerImpl` class which, from the JIRA ticket, is believed to be a legacy artifact slowing down testing originally introduced in the `ClusterScheduler` class.\n\nAuthor: Brennon York <brennon.york@capitalone.com>\n\nCloses #3851 from brennonyork/SPARK-794 and squashes the following commits:\n\n04c3e64 [Brennon York] Removed sleep() from the stop() method","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/b96008d5529bac5fd57b76554fd01760139cffff"},{"sha":"3fddc9468fa50e7683caa973fec6c52e1132268d","author":{"email":"2d6ce45e1b2962f3cef0b00c8785c555e8a958f9@outlook.com","name":"Dale"},"message":"[SPARK-4787] Stop SparkContext if a DAGScheduler init error occurs\n\nAuthor: Dale <tigerquoll@outlook.com>\n\nCloses #3809 from tigerquoll/SPARK-4787 and squashes the following commits:\n\n5661e01 [Dale] [SPARK-4787] Ensure that call to stop() doesn't lose the exception by using a finally block.\n2172578 [Dale] [SPARK-4787] Stop context properly if an exception occurs during DAGScheduler initialization.","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/3fddc9468fa50e7683caa973fec6c52e1132268d"},{"sha":"e767d7ddac5c2330af553f2a74b8575dfc7afb67","author":{"email":"65b3d949b761bb871ae7a6019f9908b0eb2edcc0@am.amrita.edu","name":"bilna"},"message":"[SPARK-4631] unit test for MQTT\n\nPlease review the unit test for MQTT\n\nAuthor: bilna <bilnap@am.amrita.edu>\nAuthor: Bilna P <bilna.p@gmail.com>\n\nCloses #3844 from Bilna/master and squashes the following commits:\n\nacea3a3 [bilna] Adding dependency with scope test\n28681fa [bilna] Merge remote-tracking branch 'upstream/master'\nfac3904 [bilna] Correction in Indentation and coding style\ned9db4c [bilna] Merge remote-tracking branch 'upstream/master'\n4b34ee7 [Bilna P] Update MQTTStreamSuite.scala\n04503cf [bilna] Added embedded broker service for mqtt test\n89d804e [bilna] Merge remote-tracking branch 'upstream/master'\nfc8eb28 [bilna] Merge remote-tracking branch 'upstream/master'\n4b58094 [Bilna P] Update MQTTStreamSuite.scala\nb1ac4ad [bilna] Added BeforeAndAfter\n5f6bfd2 [bilna] Added BeforeAndAfter\ne8b6623 [Bilna P] Update MQTTStreamSuite.scala\n5ca6691 [Bilna P] Update MQTTStreamSuite.scala\n8616495 [bilna] [SPARK-4631] unit test for MQTT","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/e767d7ddac5c2330af553f2a74b8575dfc7afb67"},{"sha":"939ba1f8f6e32fef9026cc43fce55b36e4b9bfd1","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-4835] Disable validateOutputSpecs for Spark Streaming jobs\n\nThis patch disables output spec. validation for jobs launched through Spark Streaming, since this interferes with checkpoint recovery.\n\nHadoop OutputFormats have a `checkOutputSpecs` method which performs certain checks prior to writing output, such as checking whether the output directory already exists.  SPARK-1100 added checks for FileOutputFormat, SPARK-1677 (#947) added a SparkConf configuration to disable these checks, and SPARK-2309 (#1088) extended these checks to run for all OutputFormats, not just FileOutputFormat.\n\nIn Spark Streaming, we might have to re-process a batch during checkpoint recovery, so `save` actions may be called multiple times.  In addition to `DStream`'s own save actions, users might use `transform` or `foreachRDD` and call the `RDD` and `PairRDD` save actions.  When output spec. validation is enabled, the second calls to these actions will fail due to existing output.\n\nThis patch automatically disables output spec. validation for jobs submitted by the Spark Streaming scheduler.  This is done by using Scala's `DynamicVariable` to propagate the bypass setting without having to mutate SparkConf or introduce a global variable.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #3832 from JoshRosen/SPARK-4835 and squashes the following commits:\n\n36eaf35 [Josh Rosen] Add comment explaining use of transform() in test.\n6485cf8 [Josh Rosen] Add test case in Streaming; fix bug for transform()\n7b3e06a [Josh Rosen] Remove Streaming-specific setting to undo this change; update conf. guide\nbf9094d [Josh Rosen] Revise disableOutputSpecValidation() comment to not refer to Spark Streaming.\ne581d17 [Josh Rosen] Deduplicate isOutputSpecValidationEnabled logic.\n762e473 [Josh Rosen] [SPARK-4835] Disable validateOutputSpecs for Spark Streaming jobs.","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/939ba1f8f6e32fef9026cc43fce55b36e4b9bfd1"},{"sha":"72396522bcf5303f761956658510672e4feb2845","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-5067][Core] Use '===' to compare well-defined case class\n\nA simple fix would be adding `assert(e1.appId == e2.appId)` for `SparkListenerApplicationStart`. But actually we can use `===` for well-defined case class directly. Therefore, instead of fixing this issue, I use `===` to compare those well-defined case classes (all fields have implemented a correct `equals` method, such as primitive types)\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #3886 from zsxwing/SPARK-5067 and squashes the following commits:\n\n0a51711 [zsxwing] Use '===' to compare well-defined case class","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/72396522bcf5303f761956658510672e4feb2845"},{"sha":"6c726a3fbd9cd3aa5f3a1992b2132b25eabb76a0","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-5069][Core] Fix the race condition of TaskSchedulerImpl.dagScheduler\n\nIt's not necessary to set `TaskSchedulerImpl.dagScheduler` in preStart. It's safe to set it after `initializeEventProcessActor()`.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #3887 from zsxwing/SPARK-5069 and squashes the following commits:\n\nd95894f [zsxwing] Fix the race condition of TaskSchedulerImpl.dagScheduler","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/6c726a3fbd9cd3aa5f3a1992b2132b25eabb76a0"},{"sha":"27e7f5a7237d9d64a3b2c8a030ba3e3a9a96b26c","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-5083][Core] Fix a flaky test in TaskResultGetterSuite\n\nBecause `sparkEnv.blockManager.master.removeBlock` is asynchronous, we need to make sure the block has already been removed before calling `super.enqueueSuccessfulTask`.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #3894 from zsxwing/SPARK-5083 and squashes the following commits:\n\nd97c03d [zsxwing] Fix a flaky test in TaskResultGetterSuite","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/27e7f5a7237d9d64a3b2c8a030ba3e3a9a96b26c"},{"sha":"5c506cecb933b156b2f06a688ee08c4347bf0d47","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-5074][Core] Fix a non-deterministic test failure\n\nAdd `assert(sc.listenerBus.waitUntilEmpty(WAIT_TIMEOUT_MILLIS))` to make sure `sparkListener` receive the message.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #3889 from zsxwing/SPARK-5074 and squashes the following commits:\n\ne61c198 [zsxwing] Fix a non-deterministic test failure","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/5c506cecb933b156b2f06a688ee08c4347bf0d47"},{"sha":"d3f07fd23cc26a70f44c52e24445974d4885d58a","author":{"email":"ba84dd49700e9e28f389de4642ccb026e4374342@gmail.com","name":"Varun Saxena"},"message":"[SPARK-4688] Have a single shared network timeout in Spark\n\n[SPARK-4688] Have a single shared network timeout in Spark\n\nAuthor: Varun Saxena <vsaxena.varun@gmail.com>\nAuthor: varunsaxena <vsaxena.varun@gmail.com>\n\nCloses #3562 from varunsaxena/SPARK-4688 and squashes the following commits:\n\n6e97f72 [Varun Saxena] [SPARK-4688] Single shared network timeout\ncd783a2 [Varun Saxena] SPARK-4688\nd6f8c29 [Varun Saxena] SCALA-4688\n9562b15 [Varun Saxena] SPARK-4688\na75f014 [varunsaxena] SPARK-4688\n594226c [varunsaxena] SPARK-4688","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/d3f07fd23cc26a70f44c52e24445974d4885d58a"},{"sha":"ce39b34404868de4ca51be06832169187b1aef7d","author":{"email":"ba60166985ef87d02b3f95e141f5ecd3cb447aa2@aliyun.com","name":"WangTao"},"message":"[SPARK-5057] Log message in failed askWithReply attempts\n\nhttps://issues.apache.org/jira/browse/SPARK-5057\n\nAuthor: WangTao <barneystinson@aliyun.com>\nAuthor: WangTaoTheTonic <barneystinson@aliyun.com>\n\nCloses #3875 from WangTaoTheTonic/SPARK-5057 and squashes the following commits:\n\n1503487 [WangTao] use string interpolation\n706c8a7 [WangTaoTheTonic] log more messages","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/ce39b34404868de4ca51be06832169187b1aef7d"},{"sha":"1c0e7ce056c79e1db96f85b8c56a479b8b043970","author":{"email":"580258f4dc0f7b7fba6752c3e796a6cb1bdd91fd@gmail.com","name":"Jongyoul Lee"},"message":"[SPARK-4465] runAsSparkUser doesn't affect TaskRunner in Mesos environme...\n\n...nt at all.\n\n- fixed a scope of runAsSparkUser from MesosExecutorDriver.run to MesosExecutorBackend.launchTask\n- See the Jira Issue for more details.\n\nAuthor: Jongyoul Lee <jongyoul@gmail.com>\n\nCloses #3741 from jongyoul/SPARK-4465 and squashes the following commits:\n\n46ad71e [Jongyoul Lee] [SPARK-4465] runAsSparkUser doesn't affect TaskRunner in Mesos environment at all. - Removed unused import\n3d6631f [Jongyoul Lee] [SPARK-4465] runAsSparkUser doesn't affect TaskRunner in Mesos environment at all. - Removed comments and adjusted indentations\n2343f13 [Jongyoul Lee] [SPARK-4465] runAsSparkUser doesn't affect TaskRunner in Mesos environment at all. - fixed a scope of runAsSparkUser from MesosExecutorDriver.run to MesosExecutorBackend.launchTask","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/1c0e7ce056c79e1db96f85b8c56a479b8b043970"},{"sha":"6c6f32574023b8e43a24f2081ff17e6e446de2f3","author":{"email":"05f4ff6cfd932210ddb781d81debbbf760fcab75@gmail.com","name":"freeman"},"message":"[SPARK-5089][PYSPARK][MLLIB] Fix vector convert\n\nThis is a small change addressing a potentially significant bug in how PySpark + MLlib handles non-float64 numpy arrays. The automatic conversion to `DenseVector` that occurs when passing RDDs to MLlib algorithms in PySpark should automatically upcast to float64s, but currently this wasn't actually happening. As a result, non-float64 would be silently parsed inappropriately during SerDe, yielding erroneous results when running, for example, KMeans.\n\nThe PR includes the fix, as well as a new test for the correct conversion behavior.\n\ndavies\n\nAuthor: freeman <the.freeman.lab@gmail.com>\n\nCloses #3902 from freeman-lab/fix-vector-convert and squashes the following commits:\n\n764db47 [freeman] Add a test for proper conversion behavior\n704f97e [freeman] Return array after changing type","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/6c6f32574023b8e43a24f2081ff17e6e446de2f3"},{"sha":"bbcba3a9430365640c0188e7ca6e0677d3227dd8","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-5093] Set spark.network.timeout to 120s consistently.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #3903 from rxin/timeout-120 and squashes the following commits:\n\n7c2138e [Reynold Xin] [SPARK-5093] Set spark.network.timeout to 120s consistently.","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/bbcba3a9430365640c0188e7ca6e0677d3227dd8"},{"sha":"04d55d8e8e4890d110ce5561b5c1ae608c34a7c9","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-5040][SQL] Support expressing unresolved attributes using $\"attribute name\" notation in SQL DSL.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #3862 from rxin/stringcontext-attr and squashes the following commits:\n\n9b10f57 [Reynold Xin] Rename StrongToAttributeConversionHelper\n72121af [Reynold Xin] [SPARK-5040][SQL] Support expressing unresolved attributes using $\"attribute name\" notation in SQL DSL.","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/04d55d8e8e4890d110ce5561b5c1ae608c34a7c9"},{"sha":"451546aa6d2e61e43b0c0f0669f18cfb7489e584","author":{"email":"e1a8399e36effe75a124102eb2a2bdfc82e0aab4@cloudera.com","name":"Kostas Sakellis"},"message":"SPARK-4843 [YARN] Squash ExecutorRunnableUtil and ExecutorRunnable\n\nExecutorRunnableUtil is a parent of ExecutorRunnable because of the yarn-alpha and yarn-stable split. Now that yarn-alpha is gone, this commit squashes the unnecessary hierarchy. The methods from ExecutorRunnableUtil are added as private.\n\nAuthor: Kostas Sakellis <kostas@cloudera.com>\n\nCloses #3696 from ksakellis/kostas-spark-4843 and squashes the following commits:\n\n486716f [Kostas Sakellis] Moved prepareEnvironment call to after yarnConf declaration\n470e22e [Kostas Sakellis] Fixed indentation and renamed sparkConf variable\n9b1b1c9 [Kostas Sakellis] SPARK-4843 [YARN] Squash ExecutorRunnableUtil and ExecutorRunnable","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/451546aa6d2e61e43b0c0f0669f18cfb7489e584"},{"sha":"a6394bc2c094c6c662237236c2effa2dabe67910","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-1600] Refactor FileInputStream tests to remove Thread.sleep() calls and SystemClock usage\n\nThis patch refactors Spark Streaming's FileInputStream tests to remove uses of Thread.sleep() and SystemClock, which should hopefully resolve some longstanding flakiness in these tests (see SPARK-1600).\n\nKey changes:\n\n- Modify FileInputDStream to use the scheduler's Clock instead of System.currentTimeMillis(); this allows it to be tested using ManualClock.\n- Fix a synchronization issue in ManualClock's `currentTime` method.\n- Add a StreamingTestWaiter class which allows callers to block until a certain number of batches have finished.\n- Change the FileInputStream tests so that files' modification times are manually set based off of ManualClock; this eliminates many Thread.sleep calls.\n- Update these tests to use the withStreamingContext fixture.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #3801 from JoshRosen/SPARK-1600 and squashes the following commits:\n\ne4494f4 [Josh Rosen] Address a potential race when setting file modification times\n8340bd0 [Josh Rosen] Use set comparisons for output.\n0b9c252 [Josh Rosen] Fix some ManualClock usage problems.\n1cc689f [Josh Rosen] ConcurrentHashMap -> SynchronizedMap\ndb26c3a [Josh Rosen] Use standard timeout in ScalaTest `eventually` blocks.\n3939432 [Josh Rosen] Rename StreamingTestWaiter to BatchCounter\n0b9c3a1 [Josh Rosen] Wait for checkpoint to complete\n863d71a [Josh Rosen] Remove Thread.sleep that was used to make task run slowly\nb4442c3 [Josh Rosen] batchTimeToSelectedFiles should be thread-safe\n15b48ee [Josh Rosen] Replace several TestWaiter methods w/ ScalaTest eventually.\nfffc51c [Josh Rosen] Revert \"Remove last remaining sleep() call\"\ndbb8247 [Josh Rosen] Remove last remaining sleep() call\n566a63f [Josh Rosen] Fix log message and comment typos\nda32f3f [Josh Rosen] Fix log message and comment typos\n3689214 [Josh Rosen] Merge remote-tracking branch 'origin/master' into SPARK-1600\nc8f06b1 [Josh Rosen] Remove Thread.sleep calls in FileInputStream CheckpointSuite test.\nd4f2d87 [Josh Rosen] Refactor file input stream tests to not rely on SystemClock.\ndda1403 [Josh Rosen] Add StreamingTestWaiter class.\n3c3efc3 [Josh Rosen] Synchronize `currentTime` in ManualClock\na95ddc4 [Josh Rosen] Modify FileInputDStream to use Clock class.","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/a6394bc2c094c6c662237236c2effa2dabe67910"},{"sha":"5e3ec1110495899a298313c4aa9c6c151c1f54da","author":{"email":"9e9ed5f135a87511c9684c6675d8a714b1e3898e@lab.ntt.co.jp","name":"kj-ki"},"message":"[Minor] Fix comments for GraphX 2D partitioning strategy\n\nThe sum of vertices on matrix (v0 to v11) is 12. And, I think one same block overlaps in this strategy.\n\nThis is minor PR, so I didn't file in JIRA.\n\nAuthor: kj-ki <kikushima.kenji@lab.ntt.co.jp>\n\nCloses #3904 from kj-ki/fix-partitionstrategy-comments and squashes the following commits:\n\n79829d9 [kj-ki] Fix comments for 2D partitioning.","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/5e3ec1110495899a298313c4aa9c6c151c1f54da"},{"sha":"4cba6eb42031b1a4cc3308833116ca5d9ccb1a89","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"SPARK-4159 [CORE] Maven build doesn't run JUnit test suites\n\nThis PR:\n\n- Reenables `surefire`, and copies config from `scalatest` (which is itself an old fork of `surefire`, so similar)\n- Tells `surefire` to test only Java tests\n- Enables `surefire` and `scalatest` for all children, and in turn eliminates some duplication.\n\nFor me this causes the Scala and Java tests to be run once each, it seems, as desired. It doesn't affect the SBT build but works for Maven. I still need to verify that all of the Scala tests and Java tests are being run.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #3651 from srowen/SPARK-4159 and squashes the following commits:\n\n2e8a0af [Sean Owen] Remove specialized SPARK_HOME setting for REPL, YARN tests as it appears to be obsolete\n12e4558 [Sean Owen] Append to unit-test.log instead of overwriting, so that both surefire and scalatest output is preserved. Also standardize/correct comments a bit.\ne6f8601 [Sean Owen] Reenable Java tests by reenabling surefire with config cloned from scalatest; centralize test config in the parent","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/4cba6eb42031b1a4cc3308833116ca5d9ccb1a89"},{"sha":"4108e5f36f8553bd728fd271baa69f7dfcc68d9b","author":{"email":"76654c02f3c3f38f074e5665b86fafb46a77566d@columbia.edu","name":"Travis Galoppo"},"message":"SPARK-5017 [MLlib] - Use SVD to compute determinant and inverse of covariance matrix\n\nMultivariateGaussian was calling both pinv() and det() on the covariance matrix, effectively performing two matrix decompositions.  Both values are now computed using the singular value decompositon. Both the pseudo-inverse and the pseudo-determinant are used to guard against singular matrices.\n\nAuthor: Travis Galoppo <tjg2107@columbia.edu>\n\nCloses #3871 from tgaloppo/spark-5017 and squashes the following commits:\n\n383b5b3 [Travis Galoppo] MultivariateGaussian - minor optimization in density calculation\na5b8bc5 [Travis Galoppo] Added additional points to tests in test suite. Fixed comment in MultivariateGaussian\n629d9d0 [Travis Galoppo] Moved some test values from var to val.\ndc3d0f7 [Travis Galoppo] Catch potential exception calculating pseudo-determinant. Style improvements.\nd448137 [Travis Galoppo] Added test suite for MultivariateGaussian, including test for degenerate case.\n1989be0 [Travis Galoppo] SPARK-5017 - Fixed to use SVD to compute determinant and inverse of covariance matrix.  Previous code called both pinv() and det(), effectively performing two matrix decompositions. Additionally, the pinv() implementation in Breeze is known to fail for singular matrices.\nb4415ea [Travis Galoppo] Merge branch 'spark-5017' of https://github.com/tgaloppo/spark into spark-5017\n6f11b6d [Travis Galoppo] SPARK-5017 - Use SVD to compute determinant and inverse of covariance matrix. Code was calling both det() and pinv(), effectively performing two matrix decompositions. Futhermore, Breeze pinv() currently fails for singular matrices.\nfd9784c [Travis Galoppo] SPARK-5017 - Use SVD to compute determinant and inverse of covariance matrix","distinct":true,"url":"https://api.github.com/repos/kai-zeng/spark/commits/4108e5f36f8553bd728fd271baa69f7dfcc68d9b"}]}