{"push_id":565620197,"size":182,"distinct_size":4,"ref":"refs/heads/SPARK-5203","head":"2f75b47512485a44d01244b02b2af93f4658741a","before":"b7348aca63418dea8f511bfc902a3f08ff0867ee","commits":[{"sha":"ec5b0f2cef4b30047c7f88bdc00d10b6aa308124","author":{"email":"32b88a1c10b8f403ebe009a6554c8304f557ca90@gmail.com","name":"Kannan Rajah"},"message":"[HOTFIX] Update pom.xml to pull MapR's Hadoop version 2.4.1.\n\nAuthor: Kannan Rajah <rkannan82@gmail.com>\n\nCloses #4108 from rkannan82/master and squashes the following commits:\n\neca095b [Kannan Rajah] Update pom.xml to pull MapR's Hadoop version 2.4.1.","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/ec5b0f2cef4b30047c7f88bdc00d10b6aa308124"},{"sha":"424d8c6ffff42e4231cc1088b7e69e3c0f5e6b56","author":{"email":"79478dbbc8641d1d6545f348f655464aefbda99b@intel.com","name":"jerryshao"},"message":"[SPARK-5297][Streaming] Fix Java file stream type erasure problem\n\nCurrent Java file stream doesn't support custom key/value type because of loss of type information, details can be seen in [SPARK-5297](https://issues.apache.org/jira/browse/SPARK-5297). Fix this problem by getting correct `ClassTag` from `Class[_]`.\n\nAuthor: jerryshao <saisai.shao@intel.com>\n\nCloses #4101 from jerryshao/SPARK-5297 and squashes the following commits:\n\ne022ca3 [jerryshao] Add Mima exclusion\necd61b8 [jerryshao] Fix Java fileInputStream type erasure problem","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/424d8c6ffff42e4231cc1088b7e69e3c0f5e6b56"},{"sha":"8c06a5faacfc71050461273133b9cf9a9dd8986f","author":{"email":"ba60166985ef87d02b3f95e141f5ecd3cb447aa2@aliyun.com","name":"WangTao"},"message":"[SPARK-5336][YARN]spark.executor.cores must not be less than spark.task.cpus\n\nhttps://issues.apache.org/jira/browse/SPARK-5336\n\nAuthor: WangTao <barneystinson@aliyun.com>\nAuthor: WangTaoTheTonic <barneystinson@aliyun.com>\n\nCloses #4123 from WangTaoTheTonic/SPARK-5336 and squashes the following commits:\n\n6c9676a [WangTao] Update ClientArguments.scala\n9632d3a [WangTaoTheTonic] minor comment fix\nd03d6fa [WangTaoTheTonic] import ordering should be alphabetical'\n3112af9 [WangTao] spark.executor.cores must not be less than spark.task.cpus","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/8c06a5faacfc71050461273133b9cf9a9dd8986f"},{"sha":"2eeada373e59d63b774ba92eb5d75fcd3a1cf8f4","author":{"email":"5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","name":"Sandy Ryza"},"message":"SPARK-1714. Take advantage of AMRMClient APIs to simplify logic in YarnA...\n\n...llocator\n\nThe goal of this PR is to simplify YarnAllocator as much as possible and get it up to the level of code quality we see in the rest of Spark.\n\nIn service of this, it does a few things:\n* Uses AMRMClient APIs for matching containers to requests.\n* Adds calls to AMRMClient.removeContainerRequest so that, when we use a container, we don't end up requesting it again.\n* Removes YarnAllocator's host->rack cache. YARN's RackResolver already does this caching, so this is redundant.\n* Adds tests for basic YarnAllocator functionality.\n* Breaks up the allocateResources method, which was previously nearly 300 lines.\n* A little bit of stylistic cleanup.\n* Fixes a bug that causes three times the requests to be filed when preferred host locations are given.\n\nThe patch is lossy. In particular, it loses the logic for trying to avoid containers bunching up on nodes. As I understand it, the logic that's gone is:\n\n* If, in a single response from the RM, we receive a set of containers on a node, and prefer some number of containers on that node greater than 0 but less than the number we received, give back the delta between what we preferred and what we received.\n\nThis seems like a weird way to avoid bunching E.g. it does nothing to avoid bunching when we don't request containers on particular nodes.\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #3765 from sryza/sandy-spark-1714 and squashes the following commits:\n\n32a5942 [Sandy Ryza] Muffle RackResolver logs\n74f56dd [Sandy Ryza] Fix a couple comments and simplify requestTotalExecutors\n60ea4bd [Sandy Ryza] Fix scalastyle\nca35b53 [Sandy Ryza] Simplify further\ne9cf8a6 [Sandy Ryza] Fix YarnClusterSuite\n257acf3 [Sandy Ryza] Remove locality stuff and more cleanup\n59a3c5e [Sandy Ryza] Take out rack stuff\n5f72fd5 [Sandy Ryza] Further documentation and cleanup\n89edd68 [Sandy Ryza] SPARK-1714. Take advantage of AMRMClient APIs to simplify logic in YarnAllocator","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/2eeada373e59d63b774ba92eb5d75fcd3a1cf8f4"},{"sha":"aa1e22b17b4ce885febe6970a2451c7d17d0acfb","author":{"email":"b96dbf74436b3f73db2f27c2fb7c966eb1f47360@databricks.com","name":"Reza Zadeh"},"message":"[MLlib] [SPARK-5301] Missing conversions and operations on IndexedRowMatrix and CoordinateMatrix\n\n* Transpose is missing from CoordinateMatrix (this is cheap to compute, so it should be there)\n* IndexedRowMatrix should be convertable to CoordinateMatrix (conversion added)\n\nTests for both added.\n\nAuthor: Reza Zadeh <reza@databricks.com>\n\nCloses #4089 from rezazadeh/matutils and squashes the following commits:\n\nec5238b [Reza Zadeh] Array -> Iterator to avoid temp array\n3ce0b5d [Reza Zadeh] Array -> Iterator\nbbc907a [Reza Zadeh] Use 'i' for index, and zipWithIndex\ncb10ae5 [Reza Zadeh] remove unnecessary import\na7ae048 [Reza Zadeh] Missing linear algebra utilities","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/aa1e22b17b4ce885febe6970a2451c7d17d0acfb"},{"sha":"7450a992b3b543a373c34fc4444a528954ac4b4a","author":{"email":"6cb8df8a3587c46e12c320c3f95fb11cee0096a3@stresearch.com","name":"nate.crosswhite"},"message":"[SPARK-4749] [mllib]: Allow initializing KMeans clusters using a seed\n\nThis implements the functionality for SPARK-4749 and provides units tests in Scala and PySpark\n\nAuthor: nate.crosswhite <nate.crosswhite@stresearch.com>\nAuthor: nxwhite-str <nxwhite-str@users.noreply.github.com>\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #3610 from nxwhite-str/master and squashes the following commits:\n\na2ebbd3 [nxwhite-str] Merge pull request #1 from mengxr/SPARK-4749-kmeans-seed\n7668124 [Xiangrui Meng] minor updates\nf8d5928 [nate.crosswhite] Addressing PR issues\n277d367 [nate.crosswhite] Merge remote-tracking branch 'upstream/master'\n9156a57 [nate.crosswhite] Merge remote-tracking branch 'upstream/master'\n5d087b4 [nate.crosswhite] Adding KMeans train with seed and Scala unit test\n616d111 [nate.crosswhite] Merge remote-tracking branch 'upstream/master'\n35c1884 [nate.crosswhite] Add kmeans initial seed to pyspark API","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/7450a992b3b543a373c34fc4444a528954ac4b4a"},{"sha":"3ee3ab592eee831d759c940eb68231817ad6d083","author":{"email":"9e9ed5f135a87511c9684c6675d8a714b1e3898e@lab.ntt.co.jp","name":"Kenji Kikushima"},"message":"[SPARK-5064][GraphX] Add numEdges upperbound validation for R-MAT graph generator to prevent infinite loop\n\nI looked into GraphGenerators#chooseCell, and found that chooseCell can't generate more edges than pow(2, (2 * (log2(numVertices)-1))) to make a Power-law graph. (Ex. numVertices:4 upperbound:4, numVertices:8 upperbound:16, numVertices:16 upperbound:64)\nIf we request more edges over the upperbound, rmatGraph fall into infinite loop. So, how about adding an argument validation?\n\nAuthor: Kenji Kikushima <kikushima.kenji@lab.ntt.co.jp>\n\nCloses #3950 from kj-ki/SPARK-5064 and squashes the following commits:\n\n4ee18c7 [Ankur Dave] Reword error message and add unit test\nd760bc7 [Kenji Kikushima] Add numEdges upperbound validation for R-MAT graph generator to prevent infinite loop.","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/3ee3ab592eee831d759c940eb68231817ad6d083"},{"sha":"812d3679f5f97df7b667cbc3365a49866ebc02d5","author":{"email":"c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","name":"Daoyuan Wang"},"message":"[SPARK-5244] [SQL] add coalesce() in sql parser\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #4040 from adrian-wang/coalesce and squashes the following commits:\n\n0ac8e8f [Daoyuan Wang] add coalesce() in sql parser","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/812d3679f5f97df7b667cbc3365a49866ebc02d5"},{"sha":"8361078efae7d79742d6be94cf5a15637ec860dd","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-5009] [SQL] Long keyword support in SQL Parsers\n\n* The `SqlLexical.allCaseVersions` will cause `StackOverflowException` if the key word is too long, the patch will fix that by normalizing all of the keywords in `SqlLexical`.\n* And make a unified SparkSQLParser for sharing the common code.\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #3926 from chenghao-intel/long_keyword and squashes the following commits:\n\n686660f [Cheng Hao] Support Long Keyword and Refactor the SQLParsers","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/8361078efae7d79742d6be94cf5a15637ec860dd"},{"sha":"b328ac6c8c489ef9abf850c45db5ad531da18d55","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"Revert \"[SPARK-5244] [SQL] add coalesce() in sql parser\"\n\nThis reverts commit 812d3679f5f97df7b667cbc3365a49866ebc02d5.","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/b328ac6c8c489ef9abf850c45db5ad531da18d55"},{"sha":"ba19689fe77b90052b587640c9ff325c5a892c20","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SQL] [Minor] Remove deprecated parquet tests\n\nThis PR removes the deprecated `ParquetQuerySuite`, renamed `ParquetQuerySuite2` to `ParquetQuerySuite`, and refactored changes introduced in #4115 to `ParquetFilterSuite` . It is a follow-up of #3644.\n\nNotice that test cases in the old `ParquetQuerySuite` have already been well covered by other test suites introduced in #3644.\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/apache/spark/4116)\n<!-- Reviewable:end -->\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #4116 from liancheng/remove-deprecated-parquet-tests and squashes the following commits:\n\nf73b8f9 [Cheng Lian] Removes deprecated Parquet test suite","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/ba19689fe77b90052b587640c9ff325c5a892c20"},{"sha":"3be2a887bf6107b6398e472872b22175ea4ae1f7","author":{"email":"f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","name":"wangfei"},"message":"[SPARK-4984][CORE][WEBUI] Adding a pop-up containing the full job description when it is very long\n\nIn some case the job description will be very long, such as a long sql. refer to #3718\nThis PR add a pop-up for job description when it is long.\n\n![image](https://cloud.githubusercontent.com/assets/7018048/5847400/c757cbbc-a207-11e4-891f-528821c2e68d.png)\n\n![image](https://cloud.githubusercontent.com/assets/7018048/5847409/d434b2b4-a207-11e4-8813-03a74b43d766.png)\n\nAuthor: wangfei <wangfei1@huawei.com>\n\nCloses #3819 from scwf/popup-descrip-ui and squashes the following commits:\n\nba02b83 [wangfei] address comments\na7c5e7b [wangfei] spot that it's been truncated\nfbf6162 [wangfei] Merge branch 'master' into popup-descrip-ui\n0bca96d [wangfei] remove no use val\n4b55c3b [wangfei] fix style issue\n353c6f4 [wangfei] pop up the description of job with a styled read-only text form field","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/3be2a887bf6107b6398e472872b22175ea4ae1f7"},{"sha":"9bad062268676aaa66dcbddd1e0ab7f2d7742425","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-5355] make SparkConf thread-safe\n\nThe SparkConf is not thread-safe, but is accessed by many threads. The getAll() could return parts of the configs if another thread is access it.\n\nThis PR changes SparkConf.settings to a thread-safe TrieMap.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #4143 from davies/safe-conf and squashes the following commits:\n\nf8fa1cf [Davies Liu] change to TrieMap\na1d769a [Davies Liu] make SparkConf thread-safe","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/9bad062268676aaa66dcbddd1e0ab7f2d7742425"},{"sha":"27bccc5ea9742ca166f6026033e7771c8a90cc0a","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-5202] [SQL] Add hql variable substitution support\n\nhttps://cwiki.apache.org/confluence/display/Hive/LanguageManual+VariableSubstitution\n\nThis is a block issue for the CLI user, it impacts the existed hql scripts from Hive.\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #4003 from chenghao-intel/substitution and squashes the following commits:\n\nbb41fd6 [Cheng Hao] revert the removed the implicit conversion\naf7c31a [Cheng Hao] add hql variable substitution support","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/27bccc5ea9742ca166f6026033e7771c8a90cc0a"},{"sha":"ca7910d6dd7693be2a675a0d6a6fcc9eb0aaeb5d","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-3424][MLLIB] cache point distances during k-means|| init\n\nThis PR ports the following feature implemented in #2634 by derrickburns:\n\n* During k-means|| initialization, we should cache costs (squared distances) previously computed.\n\nIt also contains the following optimization:\n\n* aggregate sumCosts directly\n* ran multiple (#runs) k-means++ in parallel\n\nI compared the performance locally on mnist-digit. Before this patch:\n\n![before](https://cloud.githubusercontent.com/assets/829644/5845647/93080862-a172-11e4-9a35-044ec711afc4.png)\n\nwith this patch:\n\n![after](https://cloud.githubusercontent.com/assets/829644/5845653/a47c29e8-a172-11e4-8e9f-08db57fe3502.png)\n\nIt is clear that each k-means|| iteration takes about the same amount of time with this patch.\n\nAuthors:\n  Derrick Burns <derrickburns@gmail.com>\n  Xiangrui Meng <meng@databricks.com>\n\nCloses #4144 from mengxr/SPARK-3424-kmeans-parallel and squashes the following commits:\n\n0a875ec [Xiangrui Meng] address comments\n4341bb8 [Xiangrui Meng] do not re-compute point distances during k-means||","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/ca7910d6dd7693be2a675a0d6a6fcc9eb0aaeb5d"},{"sha":"fcb3e1862ffe784f39bde467e8d24c1b7ed3afbb","author":{"email":"c572ebc9738c89204aa8ff4fe5a67b1b3db65d0e@gmail.com","name":"Basin"},"message":"[SPARK-5317]Set BoostingStrategy.defaultParams With Enumeration Algo.Classification or Algo.Regression\n\nJIRA Issue: https://issues.apache.org/jira/browse/SPARK-5317\nWhen setting the BoostingStrategy.defaultParams(\"Classification\"), It's more straightforward to set it with the Enumeration Algo.Classification, just like BoostingStragety.defaultParams(Algo.Classification).\nI overload the method BoostingStragety.defaultParams().\n\nAuthor: Basin <jpsachilles@gmail.com>\n\nCloses #4103 from Peishen-Jia/stragetyAlgo and squashes the following commits:\n\n87bab1c [Basin] Docs and Code documentations updated.\n3b72875 [Basin] defaultParams(algoStr: String) call defaultParams(algo: Algo).\n7c1e6ee [Basin] Doc of Java updated. algo -> algoStr instead.\nd5c8a2e [Basin] Merge branch 'stragetyAlgo' of github.com:Peishen-Jia/spark into stragetyAlgo\n65f96ce [Basin] mllib-ensembles doc modified.\ne04a5aa [Basin] boostingstrategy.defaultParam string algo to enumeration.\n68cf544 [Basin] mllib-ensembles doc modified.\na4aea51 [Basin] boostingstrategy.defaultParam string algo to enumeration.","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/fcb3e1862ffe784f39bde467e8d24c1b7ed3afbb"},{"sha":"3027f06b4127ab23a43c5ce8cebf721e3b6766e5","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-5147][Streaming] Delete the received data WAL log periodically\n\nThis is a refactored fix based on jerryshao 's PR #4037\nThis enabled deletion of old WAL files containing the received block data.\nImprovements over #4037\n- Respecting the rememberDuration of all receiver streams. In #4037, if there were two receiver streams with multiple remember durations, the deletion would have delete based on the shortest remember duration, thus deleting data prematurely for the receiver stream with longer remember duration.\n- Added unit test to test creation of receiver WAL, automatic deletion, and respecting of remember duration.\n\njerryshao I am going to merge this ASAP to make it 1.2.1 Thanks for the initial draft of this PR. Made my job much easier.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\nAuthor: jerryshao <saisai.shao@intel.com>\n\nCloses #4149 from tdas/SPARK-5147 and squashes the following commits:\n\n730798b [Tathagata Das] Added comments.\nc4cf067 [Tathagata Das] Minor fixes\n2579b27 [Tathagata Das] Refactored the fix to make sure that the cleanup respects the remember duration of all the receiver streams\n2736fd1 [jerryshao] Delete the old WAL log periodically","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/3027f06b4127ab23a43c5ce8cebf721e3b6766e5"},{"sha":"246111d179a2f3f6b97a5c2b121d8ddbfd1c9aad","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-5365][MLlib] Refactor KMeans to reduce redundant data\n\nIf a point is selected as new centers for many runs, it would collect many redundant data. This pr refactors it.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #4159 from viirya/small_refactor_kmeans and squashes the following commits:\n\n25487e6 [Liang-Chi Hsieh] Refactor codes to reduce redundant data.","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/246111d179a2f3f6b97a5c2b121d8ddbfd1c9aad"},{"sha":"820ce03597350257abe0c5c96435c555038e3e6c","author":{"email":"5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","name":"Sandy Ryza"},"message":"SPARK-5370. [YARN] Remove some unnecessary synchronization in YarnAlloca...\n\n...tor\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #4164 from sryza/sandy-spark-5370 and squashes the following commits:\n\n0c8d736 [Sandy Ryza] SPARK-5370. [YARN] Remove some unnecessary synchronization in YarnAllocator","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/820ce03597350257abe0c5c96435c555038e3e6c"},{"sha":"3c3fa632e6ba45ce536065aa1145698385301fb2","author":{"email":"79478dbbc8641d1d6545f348f655464aefbda99b@intel.com","name":"jerryshao"},"message":"[SPARK-5233][Streaming] Fix error replaying of WAL introduced bug\n\nBecause of lacking of `BlockAllocationEvent` in WAL recovery, the dangled event will mix into the new batch, which will lead to the wrong result. Details can be seen in [SPARK-5233](https://issues.apache.org/jira/browse/SPARK-5233).\n\nAuthor: jerryshao <saisai.shao@intel.com>\n\nCloses #4032 from jerryshao/SPARK-5233 and squashes the following commits:\n\nf0b0c0b [jerryshao] Further address the comments\na237c75 [jerryshao] Address the comments\ne356258 [jerryshao] Fix bug in unit test\n558bdc3 [jerryshao] Correctly replay the WAL log when recovering from failure","distinct":false,"url":"https://api.github.com/repos/guowei2/spark/commits/3c3fa632e6ba45ce536065aa1145698385301fb2"}]}