{"push_id":848914055,"size":208,"distinct_size":17,"ref":"refs/heads/cluster_multi_roles","head":"ee6ad85e04b3a7b0c7c30b19072ab311f4558205","before":"44b2ad44d12c87f66c1dbd70a6672b7bb8710e12","commits":[{"sha":"ce3f9a80657751ee0bc0ed6a9b6558acbb40af4f","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-11091] [SQL] Change spark.sql.canonicalizeView to spark.sql.nativeView.\n\nhttps://issues.apache.org/jira/browse/SPARK-11091\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #9103 from yhuai/SPARK-11091.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/ce3f9a80657751ee0bc0ed6a9b6558acbb40af4f"},{"sha":"8b32885704502ab2a715cf5142d7517181074428","author":{"email":"300d085397e665a9ec270b0f82922e78a289d85d@gmail.com","name":"Monica Liu"},"message":"[SPARK-10981] [SPARKR] SparkR Join improvements\n\nI was having issues with collect() and orderBy() in Spark 1.5.0 so I used the DataFrame.R file and test_sparkSQL.R file from the Spark 1.5.1 download. I only modified the join() function in DataFrame.R to include \"full\", \"fullouter\", \"left\", \"right\", and \"leftsemi\" and added corresponding test cases in the test for join() and merge() in test_sparkSQL.R file.\nPull request because I filed this JIRA bug report:\nhttps://issues.apache.org/jira/browse/SPARK-10981\n\nAuthor: Monica Liu <liu.monica.f@gmail.com>\n\nCloses #9029 from mfliu/master.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/8b32885704502ab2a715cf5142d7517181074428"},{"sha":"390b22fad69a33eb6daee25b6b858a2e768670a5","author":{"email":"3f27eab2afd49386f0ac811afb0bc245aa92c0fa@intel.com","name":"Sun Rui"},"message":"[SPARK-10996] [SPARKR] Implement sampleBy() in DataFrameStatFunctions.\n\nAuthor: Sun Rui <rui.sun@intel.com>\n\nCloses #9023 from sun-rui/SPARK-10996.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/390b22fad69a33eb6daee25b6b858a2e768670a5"},{"sha":"135a2ce5b0b927b512c832d61c25e7b9d57e30be","author":{"email":"a1a405b82800b492a7268cba6bc58270d2b527de@yahoo-inc.com","name":"Tom Graves"},"message":"[SPARK-10619] Can't sort columns on Executor Page\n\nshould pick into spark 1.5.2 also.\n\nhttps://issues.apache.org/jira/browse/SPARK-10619\n\nlooks like this was broken by commit: https://github.com/apache/spark/commit/fb1d06fc242ec00320f1a3049673fbb03c4a6eb9#diff-b8adb646ef90f616c34eb5c98d1ebd16\nIt looks like somethings were change to use the UIUtils.listingTable but executor page wasn't converted so when it removed sortable from the UIUtils. TABLE_CLASS_NOT_STRIPED it broke this page.\n\nSimply add the sortable tag back in and it fixes both active UI and the history server UI.\n\nAuthor: Tom Graves <tgraves@yahoo-inc.com>\n\nCloses #9101 from tgravescs/SPARK-10619.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/135a2ce5b0b927b512c832d61c25e7b9d57e30be"},{"sha":"31f315981709251d5d26c508a3dc62cf0e6f87e1","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-11040] [NETWORK] Make sure SASL handler delegates all events.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #9053 from vanzin/SPARK-11040.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/31f315981709251d5d26c508a3dc62cf0e6f87e1"},{"sha":"7e1308d37f6ca35f063e67e4b87a77e932ad89a5","author":{"email":"b1011540267ee7fe5a02243ad09c007ca1860809@us.ibm.com","name":"Huaxin Gao"},"message":"[SPARK-8386] [SQL] add write.mode for insertIntoJDBC when the parm overwrite is false\n\nthe fix is for jira https://issues.apache.org/jira/browse/SPARK-8386\n\nAuthor: Huaxin Gao <huaxing@us.ibm.com>\n\nCloses #9042 from huaxingao/spark8386.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/7e1308d37f6ca35f063e67e4b87a77e932ad89a5"},{"sha":"615cc858cf913522059b6ebdde65f0204f4fb030","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-10973]\n\nClose #9064\nClose #9063\nClose #9062\n\nThese pull requests were merged into branch-1.5, branch-1.4, and branch-1.3.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/615cc858cf913522059b6ebdde65f0204f4fb030"},{"sha":"cf2e0ae7205443f052463e8cb9334ae2b6df2d0e","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11096] Post-hoc review Netty based RPC implementation - round 2\n\nA few more changes:\n\n1. Renamed IDVerifier -> RpcEndpointVerifier\n2. Renamed NettyRpcAddress -> RpcEndpointAddress\n3. Simplified NettyRpcHandler a bit by removing the connection count tracking. This is OK because I now force spark.shuffle.io.numConnectionsPerPeer to 1\n4. Reduced spark.rpc.connect.threads to 64. It would be great to eventually remove this extra thread pool.\n5. Minor cleanup & documentation.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9112 from rxin/SPARK-11096.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/cf2e0ae7205443f052463e8cb9334ae2b6df2d0e"},{"sha":"9a430a027faafb083ca569698effb697af26a1db","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11068] [SQL] [FOLLOW-UP] move execution listener to util\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9119 from cloud-fan/callback.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/9a430a027faafb083ca569698effb697af26a1db"},{"sha":"56d7da14ab8f89bf4f303b27f51fd22d23967ffb","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-10104] [SQL] Consolidate different forms of table identifiers\n\nRight now, we have QualifiedTableName, TableIdentifier, and Seq[String] to represent table identifiers. We should only have one form and TableIdentifier is the best one because it provides methods to get table name, database name, return unquoted string, and return quoted string.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\nAuthor: Wenchen Fan <cloud0fan@163.com>\n\nCloses #8453 from cloud-fan/table-name.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/56d7da14ab8f89bf4f303b27f51fd22d23967ffb"},{"sha":"2b5e31c7e97811ef7b4da47609973b7f51444346","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11113] [SQL] Remove DeveloperApi annotation from private classes.\n\no.a.s.sql.catalyst and o.a.s.sql.execution are supposed to be private.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9121 from rxin/SPARK-11113.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/2b5e31c7e97811ef7b4da47609973b7f51444346"},{"sha":"1baaf2b9bd7c949a8f95cd14fc1be2a56e1139b3","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-10829] [SQL] Filter combine partition key and attribute doesn't work in DataSource scan\n\n```scala\nwithSQLConf(SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> \"true\") {\n      withTempPath { dir =>\n        val path = s\"${dir.getCanonicalPath}/part=1\"\n        (1 to 3).map(i => (i, i.toString)).toDF(\"a\", \"b\").write.parquet(path)\n\n        // If the \"part = 1\" filter gets pushed down, this query will throw an exception since\n        // \"part\" is not a valid column in the actual Parquet file\n        checkAnswer(\n          sqlContext.read.parquet(path).filter(\"a > 0 and (part = 0 or a > 1)\"),\n          (2 to 3).map(i => Row(i, i.toString, 1)))\n      }\n    }\n```\n\nWe expect the result to be:\n```\n2,1\n3,1\n```\nBut got\n```\n1,1\n2,1\n3,1\n```\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #8916 from chenghao-intel/partition_filter.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/1baaf2b9bd7c949a8f95cd14fc1be2a56e1139b3"},{"sha":"4ace4f8a9c91beb21a0077e12b75637a4560a542","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-11017] [SQL] Support ImperativeAggregates in TungstenAggregate\n\nThis patch extends TungstenAggregate to support ImperativeAggregate functions. The existing TungstenAggregate operator only supported DeclarativeAggregate functions, which are defined in terms of Catalyst expressions and can be evaluated via generated projections. ImperativeAggregate functions, on the other hand, are evaluated by calling their `initialize`, `update`, `merge`, and `eval` methods.\n\nThe basic strategy here is similar to how SortBasedAggregate evaluates both types of aggregate functions: use a generated projection to evaluate the expression-based declarative aggregates with dummy placeholder expressions inserted in place of the imperative aggregate function output, then invoke the imperative aggregate functions and target them against the aggregation buffer. The bulk of the diff here consists of code that was copied and adapted from SortBasedAggregate, with some key changes to handle TungstenAggregate's sort fallback path.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9038 from JoshRosen/support-interpreted-in-tungsten-agg-final.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/4ace4f8a9c91beb21a0077e12b75637a4560a542"},{"sha":"9808052b5adfed7dafd6c1b3971b998e45b2799a","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-11076] [SQL] Add decimal support for floor and ceil\n\nActually all of the `UnaryMathExpression` doens't support the Decimal, will create follow ups for supporing it. This is the first PR which will be good to review the approach I am taking.\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #9086 from chenghao-intel/ceiling.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/9808052b5adfed7dafd6c1b3971b998e45b2799a"},{"sha":"0f62c2282bb30cb4fb6eea9d28b198d557a79b22","author":{"email":"11cbff5128c5946ce85172f5628bfe117c16c968@ipcoop.com","name":"Adam Lewandowski"},"message":"[SPARK-11093] [CORE] ChildFirstURLClassLoader#getResources should return all found resources, not just those in the child classloader\n\nAuthor: Adam Lewandowski <alewandowski@ipcoop.com>\n\nCloses #9106 from alewando/childFirstFix.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/0f62c2282bb30cb4fb6eea9d28b198d557a79b22"},{"sha":"aec4400beffc569c13cceea2d0c481dfa3f34175","author":{"email":"ced9a24e2e494bb60341a9f26a61a78e7a11abad@apache.org","name":"Jeff Zhang"},"message":"[SPARK-11099] [SPARK SHELL] [SPARK SUBMIT] Default conf property file i…\n\nPlease help review it. Thanks\n\nAuthor: Jeff Zhang <zjffdu@apache.org>\n\nCloses #9114 from zjffdu/SPARK-11099.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/aec4400beffc569c13cceea2d0c481dfa3f34175"},{"sha":"523adc24a683930304f408d477607edfe9de7b76","author":{"email":"62e58dde204da426c4ca6686c39866dd1719988a@zepler.org","name":"shellberg"},"message":"[SPARK-11066] Update DAGScheduler's \"misbehaved ResultHandler\"\n\nRestrict tasks (of job) to only 1 to ensure that the causing Exception asserted for job failure is the deliberately thrown DAGSchedulerSuiteDummyException intended, not an UnsupportedOperationException from any second/subsequent tasks that can propagate from a race condition during code execution.\n\nAuthor: shellberg <sah@zepler.org>\n\nCloses #9076 from shellberg/shellberg-DAGSchedulerSuite-misbehavedResultHandlerTest-patch-1.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/523adc24a683930304f408d477607edfe9de7b76"},{"sha":"d45a0d3ca23df86cf0a95508ccc3b4b98f1b611c","author":{"email":"dbe0629cf0118f7dffec38856cad0ec7ac880251@intel.com","name":"Carson Wang"},"message":"[SPARK-11047] Internal accumulators miss the internal flag when replaying events in the history server\n\nInternal accumulators don't write the internal flag to event log. So on the history server Web UI, all accumulators are not internal. This causes incorrect peak execution memory and unwanted accumulator table displayed on the stage page.\nTo fix it, I add the \"internal\" property of AccumulableInfo when writing the event log.\n\nAuthor: Carson Wang <carson.wang@intel.com>\n\nCloses #9061 from carsonwang/accumulableBug.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/d45a0d3ca23df86cf0a95508ccc3b4b98f1b611c"},{"sha":"b591de7c07ba8e71092f71e34001520bec995a8a","author":{"email":"b9217bc121eba855e017c21a9fae7701efbccbc5@falkonry.com","name":"Nick Pritchard"},"message":"[SPARK-11039][Documentation][Web UI] Document additional ui configurations\n\nAdd documentation for configuration:\n- spark.sql.ui.retainedExecutions\n- spark.streaming.ui.retainedBatches\n\nAuthor: Nick Pritchard <nicholas.pritchard@falkonry.com>\n\nCloses #9052 from pnpritchard/SPARK-11039.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/b591de7c07ba8e71092f71e34001520bec995a8a"},{"sha":"a5719804c5ed99ce36bd0dd230ab8b3b7a3b92e3","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-11071] [LAUNCHER] Fix flakiness in LauncherServerSuite::timeout.\n\nThe test could fail depending on scheduling of the various threads\ninvolved; the change removes some sources of races, while making the\ntest a little more resilient by trying a few times before giving up.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #9079 from vanzin/SPARK-11071.","distinct":false,"url":"https://api.github.com/repos/tnachen/spark/commits/a5719804c5ed99ce36bd0dd230ab8b3b7a3b92e3"}]}