{"push_id":918849928,"size":1000,"distinct_size":1,"ref":"refs/heads/SPARK-7780-intercept-in-logisticregressionwithLBFGS-should-not-be-regularized","head":"2bf289b2ab92ff9da742d22e1feda0b57f8a796c","before":"0cedd50368eeda594eafdb9500ed162ff33f2e25","commits":[{"sha":"e1eef248f13f6c334fe4eea8a29a1de5470a2e62","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-11104] [STREAMING] Fix a deadlock in StreamingContex.stop\n\nThe following deadlock may happen if shutdownHook and StreamingContext.stop are running at the same time.\n```\nJava stack information for the threads listed above:\n===================================================\n\"Thread-2\":\n\tat org.apache.spark.streaming.StreamingContext.stop(StreamingContext.scala:699)\n\t- waiting to lock <0x00000005405a1680> (a org.apache.spark.streaming.StreamingContext)\n\tat org.apache.spark.streaming.StreamingContext.org$apache$spark$streaming$StreamingContext$$stopOnShutdown(StreamingContext.scala:729)\n\tat org.apache.spark.streaming.StreamingContext$$anonfun$start$1.apply$mcV$sp(StreamingContext.scala:625)\n\tat org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:266)\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:236)\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:236)\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:236)\n\tat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1697)\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:236)\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:236)\n\tat org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:236)\n\tat scala.util.Try$.apply(Try.scala:161)\n\tat org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:236)\n\t- locked <0x00000005405b6a00> (a org.apache.spark.util.SparkShutdownHookManager)\n\tat org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:216)\n\tat org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)\n\"main\":\n\tat org.apache.spark.util.SparkShutdownHookManager.remove(ShutdownHookManager.scala:248)\n\t- waiting to lock <0x00000005405b6a00> (a org.apache.spark.util.SparkShutdownHookManager)\n\tat org.apache.spark.util.ShutdownHookManager$.removeShutdownHook(ShutdownHookManager.scala:199)\n\tat org.apache.spark.streaming.StreamingContext.stop(StreamingContext.scala:712)\n\t- locked <0x00000005405a1680> (a org.apache.spark.streaming.StreamingContext)\n\tat org.apache.spark.streaming.StreamingContext.stop(StreamingContext.scala:684)\n\t- locked <0x00000005405a1680> (a org.apache.spark.streaming.StreamingContext)\n\tat org.apache.spark.streaming.SessionByKeyBenchmark$.main(SessionByKeyBenchmark.scala:108)\n\tat org.apache.spark.streaming.SessionByKeyBenchmark.main(SessionByKeyBenchmark.scala)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:497)\n\tat org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:680)\n\tat org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)\n\tat org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)\n\tat org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)\n\tat org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)\n```\n\nThis PR just moved `ShutdownHookManager.removeShutdownHook` out of `synchronized` to avoid deadlock.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #9116 from zsxwing/stop-deadlock.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/e1eef248f13f6c334fe4eea8a29a1de5470a2e62"},{"sha":"ac09a3a465f3b57f3964c5fd621ab0d2216e2354","author":{"email":"836fea426e23a95a8b4d10b3acb424e8ba094d6b@us.ibm.com","name":"gweidner"},"message":"[SPARK-11109] [CORE] Move FsHistoryProvider off deprecated AccessControlException\n\nSwitched from deprecated org.apache.hadoop.fs.permission.AccessControlException to org.apache.hadoop.security.AccessControlException.\n\nAuthor: gweidner <gweidner@us.ibm.com>\n\nCloses #9144 from gweidner/SPARK-11109.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/ac09a3a465f3b57f3964c5fd621ab0d2216e2354"},{"sha":"1ec0a0dc2819d3db3555799cb78c2946f652bff4","author":{"email":"011a1ffd3f2d382c543a606f9dfc7c1afcbc6988@gmail.com","name":"Bhargav Mangipudi"},"message":"[SPARK-11050] [MLLIB] PySpark SparseVector can return wrong index in e…\n\n…rror message\n\nFor negative indices in the SparseVector, we update the index value. If we have an incorrect index\nat this point, the error message has the incorrect *updated* index instead of the original one. This\nchange contains the fix for the same.\n\nAuthor: Bhargav Mangipudi <bhargav.mangipudi@gmail.com>\n\nCloses #9069 from bhargav/spark-10759.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/1ec0a0dc2819d3db3555799cb78c2946f652bff4"},{"sha":"10046ea76cf8f0d08fe7ef548e4dbec69d9c73b8","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":"[SPARK-10599] [MLLIB] Lower communication for block matrix multiplication\n\nThis PR aims to decrease communication costs in BlockMatrix multiplication in two ways:\n - Simulate the multiplication on the driver, and figure out which blocks actually need to be shuffled\n - Send the block once to a partition, and join inside the partition rather than sending multiple copies to the same partition\n\n**NOTE**: One important note is that right now, the old behavior of checking for multiple blocks with the same index is lost. This is not hard to add, but is a little more expensive than how it was.\n\nInitial benchmarking showed promising results (look below), however I did hit some `FileNotFound` exceptions with the new implementation after the shuffle.\n\nSize A: 1e5 x 1e5\nSize B: 1e5 x 1e5\nBlock Sizes: 1024 x 1024\nSparsity: 0.01\nOld implementation: 1m 13s\nNew implementation: 9s\n\ncc avulanov Would you be interested in helping me benchmark this? I used your code from the mailing list (which you sent about 3 months ago?), and the old implementation didn't even run, but the new implementation completed in 268s in a 120 GB / 16 core cluster\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #8757 from brkyvz/opt-bmm.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/10046ea76cf8f0d08fe7ef548e4dbec69d9c73b8"},{"sha":"8ac71d62d976bbfd0159cac6816dd8fa580ae1cb","author":{"email":"ffad4dc97bb49db436733c212788f8b59c50dd2b@gmail.com","name":"zero323"},"message":"[SPARK-11084] [ML] [PYTHON] Check if index can contain non-zero value before binary search\n\nAt this moment `SparseVector.__getitem__` executes `np.searchsorted` first and checks if result is in an expected range after that. It is possible to check if index can contain non-zero value before executing `np.searchsorted`.\n\nAuthor: zero323 <matthew.szymkiewicz@gmail.com>\n\nCloses #9098 from zero323/sparse_vector_getitem_improved.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/8ac71d62d976bbfd0159cac6816dd8fa580ae1cb"},{"sha":"e1e77b22b3b577909a12c3aa898eb53be02267fd","author":{"email":"66332c93c294b372ab1bdd5acb9b89a538e2d2f2@gmail.com","name":"Yuhao Yang"},"message":"[SPARK-11029] [ML] Add computeCost to KMeansModel in spark.ml\n\njira: https://issues.apache.org/jira/browse/SPARK-11029\n\nWe should add a method analogous to spark.mllib.clustering.KMeansModel.computeCost to spark.ml.clustering.KMeansModel.\nThis will be a temp fix until we have proper evaluators defined for clustering.\n\nAuthor: Yuhao Yang <hhbyyh@gmail.com>\nAuthor: yuhaoyang <yuhao@zhanglipings-iMac.local>\n\nCloses #9073 from hhbyyh/computeCost.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/e1e77b22b3b577909a12c3aa898eb53be02267fd"},{"sha":"cca2258685147be6c950c9f5c4e50eaa1e090714","author":{"email":"15c0155b091954fd32425c4e2aff73ac80b23be8@gmail.com","name":"Luvsandondov Lkhamsuren"},"message":"[SPARK-9963] [ML] RandomForest cleanup: replace predictNodeIndex with predictImpl\n\npredictNodeIndex is moved to LearningNode and renamed predictImpl for consistency with Node.predictImpl\n\nAuthor: Luvsandondov Lkhamsuren <lkhamsurenl@gmail.com>\n\nCloses #8609 from lkhamsurenl/SPARK-9963.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/cca2258685147be6c950c9f5c4e50eaa1e090714"},{"sha":"254937420678a299f06b6f4e2696c623da56cf3a","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11165] Logging trait should be private - not DeveloperApi.\n\nIts classdoc actually says; \"NOTE: DO NOT USE this class outside of Spark. It is intended as an internal utility.\"\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9155 from rxin/private-logging-trait.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/254937420678a299f06b6f4e2696c623da56cf3a"},{"sha":"57f83e36d63bbd79663c49a6c1e8f6c3c8fe4789","author":{"email":"110f47070ba3ccc2d79e32307817c523b8a06186@tresata.com","name":"Koert Kuipers"},"message":"[SPARK-10185] [SQL] Feat sql comma separated paths\n\nMake sure comma-separated paths get processed correcly in ResolvedDataSource for a HadoopFsRelationProvider\n\nAuthor: Koert Kuipers <koert@tresata.com>\n\nCloses #8416 from koertkuipers/feat-sql-comma-separated-paths.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/57f83e36d63bbd79663c49a6c1e8f6c3c8fe4789"},{"sha":"022a8f6a1f7cb477a65a65482982c021ce08a73c","author":{"email":"8df74dda184c19df19d4755602108540e071bf38@plista.com","name":"ph"},"message":"[SPARK-11129] [MESOS] Link Spark WebUI from Mesos WebUI\n\nMesos has a feature for linking to frameworks running on top of Mesos\nfrom the Mesos WebUI. This commit enables Spark to make use of this\nfeature so one can directly visit the running Spark WebUIs from the\nMesos WebUI.\n\nAuthor: ph <ph@plista.com>\n\nCloses #9135 from philipphoffmann/SPARK-11129.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/022a8f6a1f7cb477a65a65482982c021ce08a73c"},{"sha":"e2dfdbb2c0523517880138f214775f9a896f2271","author":{"email":"2c0006ca50109642a00c809a04a8dbcd53363bf2@gmail.com","name":"huangzhaowei"},"message":"[SPARK-11000] [YARN] Load `metadata.Hive` class only when `hive.metastore.uris` was set to avoid bootting the database twice\n\nAuthor: huangzhaowei <carlmartinmax@gmail.com>\n\nCloses #9026 from SaintBacchus/SPARK-11000.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/e2dfdbb2c0523517880138f214775f9a896f2271"},{"sha":"3895b2113a726171b3c9c04fe41b3cc93d6d14b5","author":{"email":"22f3e725b314afeca4d45e12683c560674029d21@gmail.com","name":"tedyu"},"message":"[SPARK-11172] Close JsonParser/Generator in test\n\nAuthor: tedyu <yuzhihong@gmail.com>\n\nCloses #9157 from tedyu/master.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/3895b2113a726171b3c9c04fe41b3cc93d6d14b5"},{"sha":"a112d69fdcd9f6d8805be6e0bc6d2211e26867c2","author":{"email":"d4728f6af9029b4fe240c80e1c28ecb547a01bc3@gmail.com","name":"Lukasz Piepiora"},"message":"[SPARK-11174] [DOCS] Fix typo in the GraphX programming guide\n\nThis patch fixes a small typo in the GraphX programming guide\n\nAuthor: Lukasz Piepiora <lpiepiora@gmail.com>\n\nCloses #9160 from lpiepiora/11174-fix-typo-in-graphx-programming-guide.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/a112d69fdcd9f6d8805be6e0bc6d2211e26867c2"},{"sha":"0480d6ca83d170618fa6a817ad64a2872438d47f","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11169] Remove the extra spaces in merge script\n\nOur merge script now turns\n```\n[SPARK-1234][SPARK-1235][SPARK-1236][SQL] description\n```\ninto\n```\n[SPARK-1234] [SPARK-1235] [SPARK-1236] [SQL] description\n```\nThe extra spaces are more annoying in git since the first line of a git commit is supposed to be very short.\n\nDoctest passes with the following command:\n```\npython -m doctest merge_spark_pr.py\n```\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9156 from rxin/SPARK-11169.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/0480d6ca83d170618fa6a817ad64a2872438d47f"},{"sha":"8d4449c7f5d528410306c288a042c4594b81a881","author":{"email":"cbb7353e6d953ef360baf960c122346276c6e320@databricks.com","name":"Patrick Wendell"},"message":"MAINTENANCE: Automated closing of pull requests.\n\nThis commit exists to close the following pull requests on Github:\n\nCloses #8737 (close requested by 'srowen')\nCloses #5323 (close requested by 'JoshRosen')\nCloses #6148 (close requested by 'JoshRosen')\nCloses #7557 (close requested by 'JoshRosen')\nCloses #7047 (close requested by 'srowen')\nCloses #8713 (close requested by 'marmbrus')\nCloses #5834 (close requested by 'srowen')\nCloses #7467 (close requested by 'tdas')\nCloses #8943 (close requested by 'xiaowen147')\nCloses #4434 (close requested by 'JoshRosen')\nCloses #8949 (close requested by 'srowen')\nCloses #5368 (close requested by 'JoshRosen')\nCloses #8186 (close requested by 'marmbrus')\nCloses #5147 (close requested by 'JoshRosen')","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/8d4449c7f5d528410306c288a042c4594b81a881"},{"sha":"a337c235a12d4ea6a7d6db457acc6b32f1915241","author":{"email":"42f1c6a8b4b7edc3dc0b2517f798bae0e044b713@thehumangeo.com","name":"Mahmoud Lababidi"},"message":"[SPARK-11158][SQL] Modified _verify_type() to be more informative on Errors by presenting the Object\n\nThe _verify_type() function had Errors that were raised when there were Type conversion issues but left out the Object in question. The Object is now added in the Error to reduce the strain on the user to debug through to figure out the Object that failed the Type conversion.\n\nThe use case for me was a Pandas DataFrame that contained 'nan' as values for columns of Strings.\n\nAuthor: Mahmoud Lababidi <mahmoud@thehumangeo.com>\nAuthor: Mahmoud Lababidi <lababidi@gmail.com>\n\nCloses #9149 from lababidi/master.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/a337c235a12d4ea6a7d6db457acc6b32f1915241"},{"sha":"94c8fef296e5cdac9a93ed34acc079e51839caa7","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-11126][SQL] Fix a memory leak in SQLListener._stageIdToStageMetrics\n\nSQLListener adds all stage infos to `_stageIdToStageMetrics`, but only removes stage infos belonging to SQL executions. This PR fixed it by ignoring stages that don't belong to SQL executions.\n\nReported by Terry Hoo in https://www.mail-archive.com/userspark.apache.org/msg38810.html\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #9132 from zsxwing/SPARK-11126.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/94c8fef296e5cdac9a93ed34acc079e51839caa7"},{"sha":"d3180c25d8cf0899a7238e7d24b35c5ae918cc1d","author":{"email":"eaca3ca4b51928dd768a043ece15380fadcb1b95@capitalone.com","name":"Brennon York"},"message":"[SPARK-7018][BUILD] Refactor dev/run-tests-jenkins into Python\n\nThis commit refactors the `run-tests-jenkins` script into Python. This refactoring was done by brennonyork in #7401; this PR contains a few minor edits from joshrosen in order to bring it up to date with other recent changes.\n\nFrom the original PR description (by brennonyork):\n\nCurrently a few things are left out that, could and I think should, be smaller JIRA's after this.\n\n1. There are still a few areas where we use environment variables where we don't need to (like `CURRENT_BLOCK`). I might get around to fixing this one in lieu of everything else, but wanted to point that out.\n2. The PR tests are still written in bash. I opted to not change those and just rewrite the runner into Python. This is a great follow-on JIRA IMO.\n3. All of the linting scripts are still in bash as well and would likely do to just add those in as follow-on JIRA's as well.\n\nCloses #7401.\n\nAuthor: Brennon York <brennon.york@capitalone.com>\n\nCloses #9161 from JoshRosen/run-tests-jenkins-refactoring.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/d3180c25d8cf0899a7238e7d24b35c5ae918cc1d"},{"sha":"beb8bc1ea588b7f9ab7effff707c0f784421364d","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-11126][SQL] Fix the potential flaky test\n\nThe unit test added in #9132 is flaky. This is a follow up PR to add `listenerBus.waitUntilEmpty` to fix it.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #9163 from zsxwing/SPARK-11126-follow-up.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/beb8bc1ea588b7f9ab7effff707c0f784421364d"},{"sha":"bd64c2d550c36405f9be25a5c6a8eaa54bf4e7e7","author":{"email":"5cafd55dd0ca8119e8e1d62a6779a4b9bbdd477d@deepsense.io","name":"Jacek Laskowski"},"message":"[SPARK-10921][YARN] Completely remove the use of SparkContext.prefer…\n\n…redNodeLocationData\n\nAuthor: Jacek Laskowski <jacek.laskowski@deepsense.io>\n\nCloses #8976 from jaceklaskowski/SPARK-10921.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/bd64c2d550c36405f9be25a5c6a8eaa54bf4e7e7"}]}