{"push_id":734680950,"size":17,"distinct_size":14,"ref":"refs/heads/sort","head":"9e2c1f69f43fe4c99e9fc996f4d56a1e532ba409","before":"0f9b6daacf35ad38cd971bdd454f0bfa2b075e00","commits":[{"sha":"63f4bcc73f5a09c1790cc3c333f08b18609de6a4","author":{"email":"bc4a65f227ddcd81bb538c67e810a3e11696e8bf@gmail.com","name":"Yu ISHIKAWA"},"message":"[SPARK-9121] [SPARKR] Get rid of the warnings about `no visible global function definition` in SparkR\n\n[[SPARK-9121] Get rid of the warnings about `no visible global function definition` in SparkR - ASF JIRA](https://issues.apache.org/jira/browse/SPARK-9121)\n\n## The Result of `dev/lint-r`\n[The result of lint-r for SPARK-9121 at the revision:1ddd0f2f1688560f88470e312b72af04364e2d49 when I have sent a PR](https://gist.github.com/yu-iskw/6f55953425901725edf6)\n\nAuthor: Yu ISHIKAWA <yuu.ishikawa@gmail.com>\n\nCloses #7567 from yu-iskw/SPARK-9121 and squashes the following commits:\n\nc8cfd63 [Yu ISHIKAWA] Fix the typo\nb1f19ed [Yu ISHIKAWA] Add a validate statement for local SparkR\n1a03987 [Yu ISHIKAWA] Load the `testthat` package in `dev/lint-r.R`, instead of using the full path of function.\n3a5e0ab [Yu ISHIKAWA] [SPARK-9121][SparkR] Get rid of the warnings about `no visible global function definition` in SparkR","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/63f4bcc73f5a09c1790cc3c333f08b18609de6a4"},{"sha":"f4785f5b82c57bce41d3dc26ed9e3c9e794c7558","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-9232] [SQL] Duplicate code in JSONRelation\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #7576 from andrewor14/clean-up-json-relation and squashes the following commits:\n\nea80803 [Andrew Or] Clean up duplicate code","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/f4785f5b82c57bce41d3dc26ed9e3c9e794c7558"},{"sha":"c03299a18b4e076cabb4b7833a1e7632c5c0dabe","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-4233] [SPARK-4367] [SPARK-3947] [SPARK-3056] [SQL] Aggregation Improvement\n\nThis is the first PR for the aggregation improvement, which is tracked by https://issues.apache.org/jira/browse/SPARK-4366 (umbrella JIRA). This PR contains work for its subtasks, SPARK-3056, SPARK-3947, SPARK-4233, and SPARK-4367.\n\nThis PR introduces a new code path for evaluating aggregate functions. This code path is guarded by `spark.sql.useAggregate2` and by default the value of this flag is true.\n\nThis new code path contains:\n* A new aggregate function interface (`AggregateFunction2`) and 7 built-int aggregate functions based on this new interface (`AVG`, `COUNT`, `FIRST`, `LAST`, `MAX`, `MIN`, `SUM`)\n* A UDAF interface (`UserDefinedAggregateFunction`) based on the new code path and two example UDAFs (`MyDoubleAvg` and `MyDoubleSum`).\n* A sort-based aggregate operator (`Aggregate2Sort`) for the new aggregate function interface .\n* A sort-based aggregate operator (`FinalAndCompleteAggregate2Sort`) for distinct aggregations (for distinct aggregations the query plan will use `Aggregate2Sort` and `FinalAndCompleteAggregate2Sort` together).\n\nWith this change, `spark.sql.useAggregate2` is `true`, the flow of compiling an aggregation query is:\n1. Our analyzer looks up functions and returns aggregate functions built based on the old aggregate function interface.\n2. When our planner is compiling the physical plan, it tries try to convert all aggregate functions to the ones built based on the new interface. The planner will fallback to the old code path if any of the following two conditions is true:\n* code-gen is disabled.\n* there is any function that cannot be converted (right now, Hive UDAFs).\n* the schema of grouping expressions contain any complex data type.\n* There are multiple distinct columns.\n\nRight now, the new code path handles a single distinct column in the query (you can have multiple aggregate functions using that distinct column). For a query having a aggregate function with DISTINCT and regular aggregate functions, the generated plan will do partial aggregations for those regular aggregate function.\n\nThanks chenghao-intel for his initial work on it.\n\nAuthor: Yin Huai <yhuai@databricks.com>\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #7458 from yhuai/UDAF and squashes the following commits:\n\n7865f5e [Yin Huai] Put the catalyst expression in the comment of the generated code for it.\nb04d6c8 [Yin Huai] Remove unnecessary change.\nf1d5901 [Yin Huai] Merge remote-tracking branch 'upstream/master' into UDAF\n35b0520 [Yin Huai] Use semanticEquals to replace grouping expressions in the output of the aggregate operator.\n3b43b24 [Yin Huai] bug fix.\n00eb298 [Yin Huai] Make it compile.\na3ca551 [Yin Huai] Merge remote-tracking branch 'upstream/master' into UDAF\ne0afca3 [Yin Huai] Gracefully fallback to old aggregation code path.\n8a8ac4a [Yin Huai] Merge remote-tracking branch 'upstream/master' into UDAF\n88c7d4d [Yin Huai] Enable spark.sql.useAggregate2 by default for testing purpose.\ndc96fd1 [Yin Huai] Many updates:\n85c9c4b [Yin Huai] newline.\n43de3de [Yin Huai] Merge remote-tracking branch 'upstream/master' into UDAF\nc3614d7 [Yin Huai] Handle single distinct column.\n68b8ee9 [Yin Huai] Support single distinct column set. WIP\n3013579 [Yin Huai] Format.\nd678aee [Yin Huai] Remove AggregateExpressionSuite.scala since our built-in aggregate functions will be based on AlgebraicAggregate and we need to have another way to test it.\ne243ca6 [Yin Huai] Add aggregation iterators.\na101960 [Yin Huai] Change MyJavaUDAF to MyDoubleSum.\n594cdf5 [Yin Huai] Change existing AggregateExpression to AggregateExpression1 and add an AggregateExpression as the common interface for both AggregateExpression1 and AggregateExpression2.\n380880f [Yin Huai] Merge remote-tracking branch 'upstream/master' into UDAF\n0a827b3 [Yin Huai] Add comments and doc. Move some classes to the right places.\na19fea6 [Yin Huai] Add UDAF interface.\n262d4c4 [Yin Huai] Make it compile.\nb2e358e [Yin Huai] Merge remote-tracking branch 'upstream/master' into UDAF\n6edb5ac [Yin Huai] Format update.\n70b169c [Yin Huai] Remove groupOrdering.\n4721936 [Yin Huai] Add CheckAggregateFunction to extendedCheckRules.\nd821a34 [Yin Huai] Cleanup.\n32aea9c [Yin Huai] Merge remote-tracking branch 'upstream/master' into UDAF\n5b46d41 [Yin Huai] Bug fix.\naff9534 [Yin Huai] Make Aggregate2Sort work with both algebraic AggregateFunctions and non-algebraic AggregateFunctions.\n2857b55 [Yin Huai] Merge remote-tracking branch 'upstream/master' into UDAF\n4435f20 [Yin Huai] Add ConvertAggregateFunction to HiveContext's analyzer.\n1b490ed [Michael Armbrust] make hive test\n8cfa6a9 [Michael Armbrust] add test\n1b0bb3f [Yin Huai] Do not bind references in AlgebraicAggregate and use code gen for all places.\n072209f [Yin Huai] Bug fix: Handle expressions in grouping columns that are not attribute references.\nf7d9e54 [Michael Armbrust] Merge remote-tracking branch 'apache/master' into UDAF\n39ee975 [Yin Huai] Code cleanup: Remove unnecesary AttributeReferences.\nb7720ba [Yin Huai] Add an analysis rule to convert aggregate function to the new version.\n5c00f3f [Michael Armbrust] First draft of codegen\n6bbc6ba [Michael Armbrust] now with correct answers\\!\nf7996d0 [Michael Armbrust] Add AlgebraicAggregate\ndded1c5 [Yin Huai] wip","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/c03299a18b4e076cabb4b7833a1e7632c5c0dabe"},{"sha":"b55a36bc30a628d76baa721d38789fc219eccc27","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-9254] [BUILD] [HOTFIX] sbt-launch-lib.bash should support HTTP/HTTPS redirection\n\nTarget file(s) can be hosted on CDN nodes. HTTP/HTTPS redirection must be supported to download these files.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #7597 from liancheng/spark-9254 and squashes the following commits:\n\nfd266ca [Cheng Lian] Uses `--fail' to make curl return non-zero value and remove garbage output when the download fails\na7cbfb3 [Cheng Lian] Supports HTTP/HTTPS redirection","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/b55a36bc30a628d76baa721d38789fc219eccc27"},{"sha":"76520955fddbda87a5c53d0a394dedc91dce67e8","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-9082] [SQL] Filter using non-deterministic expressions should not be pushed down\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #7446 from cloud-fan/filter and squashes the following commits:\n\n330021e [Wenchen Fan] add exists to tree node\n2cab68c [Wenchen Fan] more enhance\n949be07 [Wenchen Fan] push down part of predicate if possible\n3912f84 [Wenchen Fan] address comments\n8ce15ca [Wenchen Fan] fix bug\n557158e [Wenchen Fan] Filter using non-deterministic expressions should not be pushed down","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/76520955fddbda87a5c53d0a394dedc91dce67e8"},{"sha":"86f80e2b4759e574fe3eb91695f81b644db87242","author":{"email":"348ff379544b45aa3f16e53a543620141423a0f6@gmail.com","name":"Yijie Shen"},"message":"[SPARK-9165] [SQL] codegen for CreateArray, CreateStruct and CreateNamedStruct\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-9165\n\nAuthor: Yijie Shen <henry.yijieshen@gmail.com>\n\nCloses #7537 from yjshen/array_struct_codegen and squashes the following commits:\n\n3a6dce6 [Yijie Shen] use infix notion in createArray test\n5e90f0a [Yijie Shen] resolve comments: classOf\n39cefb8 [Yijie Shen] codegen for createArray createStruct & createNamedStruct","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/86f80e2b4759e574fe3eb91695f81b644db87242"},{"sha":"e0b7ba59a1ace9b78a1ad6f3f07fe153db20b52c","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-9024] Unsafe HashJoin/HashOuterJoin/HashSemiJoin\n\nThis PR introduce unsafe version (using UnsafeRow) of HashJoin, HashOuterJoin and HashSemiJoin, including the broadcast one and shuffle one (except FullOuterJoin, which is better to be implemented using SortMergeJoin).\n\nIt use HashMap to store UnsafeRow right now, will change to use BytesToBytesMap for better performance (in another PR).\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #7480 from davies/unsafe_join and squashes the following commits:\n\n6294b1e [Davies Liu] fix projection\n10583f1 [Davies Liu] Merge branch 'master' of github.com:apache/spark into unsafe_join\ndede020 [Davies Liu] fix test\n84c9807 [Davies Liu] address comments\na05b4f6 [Davies Liu] support UnsafeRow in LeftSemiJoinBNL and BroadcastNestedLoopJoin\n611d2ed [Davies Liu] Merge branch 'master' of github.com:apache/spark into unsafe_join\n9481ae8 [Davies Liu] return UnsafeRow after join()\nca2b40f [Davies Liu] revert unrelated change\n68f5cd9 [Davies Liu] Merge branch 'master' of github.com:apache/spark into unsafe_join\n0f4380d [Davies Liu] ada a comment\n69e38f5 [Davies Liu] Merge branch 'master' of github.com:apache/spark into unsafe_join\n1a40f02 [Davies Liu] refactor\nab1690f [Davies Liu] address comments\n60371f2 [Davies Liu] use UnsafeRow in SemiJoin\na6c0b7d [Davies Liu] Merge branch 'master' of github.com:apache/spark into unsafe_join\n184b852 [Davies Liu] fix style\n6acbb11 [Davies Liu] fix tests\n95d0762 [Davies Liu] remove println\nbea4a50 [Davies Liu] Unsafe HashJoin","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/e0b7ba59a1ace9b78a1ad6f3f07fe153db20b52c"},{"sha":"8486cd853104255b4eb013860bba793eef4e74e7","author":{"email":"9835e449f9a666ddd8ba8aeb460af39a5130691b@databricks.com","name":"Feynman Liang"},"message":"[SPARK-9224] [MLLIB] OnlineLDA Performance Improvements\n\nIn-place updates, reduce number of transposes, and vectorize operations in OnlineLDA implementation.\n\nAuthor: Feynman Liang <fliang@databricks.com>\n\nCloses #7454 from feynmanliang/OnlineLDA-perf-improvements and squashes the following commits:\n\n78b0f5a [Feynman Liang] Make in-place variables vals, fix BLAS error\n7f62a55 [Feynman Liang] --amend\nc62cb1e [Feynman Liang] Outer product for stats, revert Range slicing\naead650 [Feynman Liang] Range slice, in-place update, reduce transposes","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/8486cd853104255b4eb013860bba793eef4e74e7"},{"sha":"cf21d05f8b5fae52b118fb8846f43d6fda1aea41","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-4366] [SQL] [Follow-up] Fix SqlParser compiling warning.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #7588 from yhuai/SPARK-4366-update1 and squashes the following commits:\n\n25f5f36 [Yin Huai] Fix SqlParser Warning.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/cf21d05f8b5fae52b118fb8846f43d6fda1aea41"},{"sha":"1aca9c13c144fa336af6afcfa666128bf77c49d4","author":{"email":"9835e449f9a666ddd8ba8aeb460af39a5130691b@databricks.com","name":"Feynman Liang"},"message":"[SPARK-8536] [MLLIB] Generalize OnlineLDAOptimizer to asymmetric document-topic Dirichlet priors\n\nModify `LDA` to take asymmetric document-topic prior distributions and `OnlineLDAOptimizer` to use the asymmetric prior during variational inference.\n\nThis PR only generalizes `OnlineLDAOptimizer` and the associated `LocalLDAModel`; `EMLDAOptimizer` and `DistributedLDAModel` still only support symmetric `alpha` (checked during `EMLDAOptimizer.initialize`).\n\nAuthor: Feynman Liang <fliang@databricks.com>\n\nCloses #7575 from feynmanliang/SPARK-8536-LDA-asymmetric-priors and squashes the following commits:\n\naf8fbb7 [Feynman Liang] Fix merge errors\nef5821d [Feynman Liang] Merge remote-tracking branch 'apache/master' into SPARK-8536-LDA-asymmetric-priors\n58f1d7b [Feynman Liang] Fix from review feedback\na6dcf70 [Feynman Liang] Change docConcentration interface and move LDAOptimizer validation to initialize, add sad path tests\n72038ff [Feynman Liang] Add tests referenced against gensim\nd4284fa [Feynman Liang] Generalize OnlineLDA to asymmetric priors, no tests","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/1aca9c13c144fa336af6afcfa666128bf77c49d4"},{"sha":"fe26584a1f5b472fb2e87aa7259aec822a619a3b","author":{"email":"c481c5e387de8107aa9fe99c851e11e1ff44d0c3@databricks.com","name":"Matei Zaharia"},"message":"[SPARK-9244] Increase some memory defaults\n\nThere are a few memory limits that people hit often and that we could\nmake higher, especially now that memory sizes have grown.\n\n- spark.akka.frameSize: This defaults at 10 but is often hit for map\n  output statuses in large shuffles. This memory is not fully allocated\n  up-front, so we can just make this larger and still not affect jobs\n  that never sent a status that large. We increase it to 128.\n\n- spark.executor.memory: Defaults at 512m, which is really small. We\n  increase it to 1g.\n\nAuthor: Matei Zaharia <matei@databricks.com>\n\nCloses #7586 from mateiz/configs and squashes the following commits:\n\nce0038a [Matei Zaharia] [SPARK-9244] Increase some memory defaults","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/fe26584a1f5b472fb2e87aa7259aec822a619a3b"},{"sha":"798dff7b4baa952c609725b852bcb6a9c9e5a317","author":{"email":"53fdba43521c8d2c2b4ff3a6f81e9f27ad86e7ac@gmail.com","name":"Iulian Dragos"},"message":"[SPARK-8975] [STREAMING] Adds a mechanism to send a new rate from the driver to the block generator\n\nFirst step for [SPARK-7398](https://issues.apache.org/jira/browse/SPARK-7398).\n\ntdas huitseeker\n\nAuthor: Iulian Dragos <jaguarul@gmail.com>\nAuthor: François Garillot <francois@garillot.net>\n\nCloses #7471 from dragos/topic/streaming-bp/dynamic-rate and squashes the following commits:\n\n8941cf9 [Iulian Dragos] Renames and other nitpicks.\n162d9e5 [Iulian Dragos] Use Reflection for accessing truly private `executor` method and use the listener bus to know when receivers have registered (`onStart` is called before receivers have registered, leading to flaky behavior).\n210f495 [Iulian Dragos] Revert \"Added a few tests that measure the receiver’s rate.\"\n0c51959 [Iulian Dragos] Added a few tests that measure the receiver’s rate.\n261a051 [Iulian Dragos] - removed field to hold the current rate limit in rate limiter - made rate limit a Long and default to Long.MaxValue (consequence of the above) - removed custom `waitUntil` and replaced it by `eventually`\ncd1397d [Iulian Dragos] Add a test for the propagation of a new rate limit from driver to receivers.\n6369b30 [Iulian Dragos] Merge pull request #15 from huitseeker/SPARK-8975\nd15de42 [François Garillot] [SPARK-8975][Streaming] Adds Ratelimiter unit tests w.r.t. spark.streaming.receiver.maxRate\n4721c7d [François Garillot] [SPARK-8975][Streaming] Add a mechanism to send a new rate from the driver to the block generator","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/798dff7b4baa952c609725b852bcb6a9c9e5a317"},{"sha":"430cd7815dc7875edd126af4b90752ba8a380cf2","author":{"email":"50f3f01caa053693ce619d596e14b0ff3901ab49@kenichimaehashi.com","name":"Kenichi Maehashi"},"message":"[SPARK-9180] fix spark-shell to accept --name option\n\nThis patch fixes [[SPARK-9180]](https://issues.apache.org/jira/browse/SPARK-9180).\nUsers can now set the app name of spark-shell using `spark-shell --name \"whatever\"`.\n\nAuthor: Kenichi Maehashi <webmaster@kenichimaehashi.com>\n\nCloses #7512 from kmaehashi/fix-spark-shell-app-name and squashes the following commits:\n\ne24991a [Kenichi Maehashi] use setIfMissing instead of setAppName\n18aa4ad [Kenichi Maehashi] fix spark-shell to accept --name option","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/430cd7815dc7875edd126af4b90752ba8a380cf2"},{"sha":"5307c9d3f7a35c0276b72e743e3a62a44d2bd0f5","author":{"email":"8757c4a255384de9fc2175f17a84f948e7081d30@gmail.com","name":"MechCoder"},"message":"[SPARK-9223] [PYSPARK] [MLLIB] Support model save/load in LDA\n\nSince save / load has been merged in LDA, it takes no time to write the wrappers in Python as well.\n\nAuthor: MechCoder <manojkumarsivaraj334@gmail.com>\n\nCloses #7587 from MechCoder/python_lda_save_load and squashes the following commits:\n\nc8e4ea7 [MechCoder] [SPARK-9223] [PySpark] Support model save/load in LDA","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/5307c9d3f7a35c0276b72e743e3a62a44d2bd0f5"},{"sha":"a721ee52705100dbd7852f80f92cde4375517e48","author":{"email":"e17b8b4554681689b2d1271ab1e80fcf2f1f56ad@email.cz","name":"martinzapletal"},"message":"[SPARK-8484] [ML] Added TrainValidationSplit for hyper-parameter tuning.\n\n- [X] Added TrainValidationSplit for hyper-parameter tuning. It randomly splits the input dataset into train and validation and use evaluation metric on the validation set to select the best model. It should be similar to CrossValidator, but simpler and less expensive.\n- [X] Simplified replacement of https://github.com/apache/spark/pull/6996\n\nAuthor: martinzapletal <zapletal-martin@email.cz>\n\nCloses #7337 from zapletal-martin/SPARK-8484-TrainValidationSplit and squashes the following commits:\n\ncafc949 [martinzapletal] Review comments https://github.com/apache/spark/pull/7337.\n511b398 [martinzapletal] Merge remote-tracking branch 'upstream/master' into SPARK-8484-TrainValidationSplit\nf4fc9c4 [martinzapletal] SPARK-8484 Resolved feedback to https://github.com/apache/spark/pull/7337\n00c4f5a [martinzapletal] SPARK-8484. Styling.\nd699506 [martinzapletal] SPARK-8484. Styling.\n93ed2ee [martinzapletal] Styling.\n3bc1853 [martinzapletal] SPARK-8484. Styling.\n2aa6f43 [martinzapletal] SPARK-8484. Added TrainValidationSplit for hyper-parameter tuning. It randomly splits the input dataset into train and validation and use evaluation metric on the validation set to select the best model.\n21662eb [martinzapletal] SPARK-8484. Added TrainValidationSplit for hyper-parameter tuning. It randomly splits the input dataset into train and validation and use evaluation metric on the validation set to select the best model.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/a721ee52705100dbd7852f80f92cde4375517e48"},{"sha":"03f851769be54f19888ffd4a94ab183812c990a7","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"do not order by expressions which still need evaluation","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/03f851769be54f19888ffd4a94ab183812c990a7"},{"sha":"9e2c1f69f43fe4c99e9fc996f4d56a1e532ba409","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"add more comments","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/9e2c1f69f43fe4c99e9fc996f4d56a1e532ba409"}]}