{"push_id":693537963,"size":683,"distinct_size":683,"ref":"refs/heads/master","head":"c19c78577a211eefe1112ebd4670a4ce7c3cc3be","before":"ee374e89cd1f08730fed9d50b742627d5b19d241","commits":[{"sha":"47728db7cfac995d9417cdf0e16d07391aabd581","author":{"email":"5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","name":"Sandy Ryza"},"message":"[SPARK-5888] [MLLIB] Add OneHotEncoder as a Transformer\n\nThis patch adds a one hot encoder for categorical features.  Planning to add documentation and another test after getting feedback on the approach.\n\nA couple choices made here:\n* There's an `includeFirst` option which, if false, creates numCategories - 1 columns and, if true, creates numCategories columns.  The default is true, which is the behavior in scikit-learn.\n* The user is expected to pass a `Seq` of category names when instantiating a `OneHotEncoder`.  These can be easily gotten from a `StringIndexer`.  The names are used for the output column names, which take the form colName_categoryName.\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #5500 from sryza/sandy-spark-5888 and squashes the following commits:\n\nf383250 [Sandy Ryza] Infer label names automatically\n6e257b9 [Sandy Ryza] Review comments\n7c539cf [Sandy Ryza] Vector transformers\n1c182dd [Sandy Ryza] SPARK-5888. [MLLIB]. Add OneHotEncoder as a Transformer","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/47728db7cfac995d9417cdf0e16d07391aabd581"},{"sha":"489700c809a7c0a836538f3d0bd58bed609e8768","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-6939] [STREAMING] [WEBUI] Add timeline and histogram graphs for streaming statistics\n\nThis is the initial work of SPARK-6939. Not yet ready for code review. Here are the screenshots:\n\n![graph1](https://cloud.githubusercontent.com/assets/1000778/7165766/465942e0-e3dc-11e4-9b05-c184b09d75dc.png)\n\n![graph2](https://cloud.githubusercontent.com/assets/1000778/7165779/53f13f34-e3dc-11e4-8714-a4a75b7e09ff.png)\n\nTODOs:\n- [x] Display more information on mouse hover\n- [x] Align the timeline and distribution graphs\n- [x] Clean up the codes\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #5533 from zsxwing/SPARK-6939 and squashes the following commits:\n\n9f7cd19 [zsxwing] Merge branch 'master' into SPARK-6939\ndeacc3f [zsxwing] Remove unused import\ncd03424 [zsxwing] Fix .rat-excludes\n70cc87d [zsxwing] Streaming Scheduling Delay => Scheduling Delay\nd457277 [zsxwing] Fix UIUtils in BatchPage\nb3f303e [zsxwing] Add comments for unclear classes and methods\nff0bff8 [zsxwing] Make InputDStream.name private[streaming]\ncc392c5 [zsxwing] Merge branch 'master' into SPARK-6939\ne275e23 [zsxwing] Move time related methods to Streaming's UIUtils\nd5d86f6 [zsxwing] Fix incorrect lastErrorTime\n3be4b7a [zsxwing] Use InputInfo\nb50fa32 [zsxwing] Jump to the batch page when clicking a point in the timeline graphs\n203605d [zsxwing] Merge branch 'master' into SPARK-6939\n74307cf [zsxwing] Reuse the data for histogram graphs to reduce the page size\n2586916 [zsxwing] Merge branch 'master' into SPARK-6939\n70d8533 [zsxwing] Remove BatchInfo.numRecords and a few renames\n7bbdc0a [zsxwing] Hide the receiver sub table if no receiver\na2972e9 [zsxwing] Add some ui tests for StreamingPage\nfd03ad0 [zsxwing] Add a test to verify no memory leak\n4a8f886 [zsxwing] Merge branch 'master' into SPARK-6939\n18607a1 [zsxwing] Merge branch 'master' into SPARK-6939\nd0b0aec [zsxwing] Clean up the codes\na459f49 [zsxwing] Add a dash line to processing time graphs\n8e4363c [zsxwing] Prepare for the demo\nc81a1ee [zsxwing] Change time unit in the graphs automatically\n4c0b43f [zsxwing] Update Streaming UI\n04c7500 [zsxwing] Make the server and client use the same timezone\nfed8219 [zsxwing] Move the x axis at the top and show a better tooltip\nc23ce10 [zsxwing] Make two graphs close\nd78672a [zsxwing] Make the X axis use the same range\n881c907 [zsxwing] Use histogram for distribution\n5688702 [zsxwing] Fix the unit test\nddf741a [zsxwing] Fix the unit test\nad93295 [zsxwing] Remove unnecessary codes\na0458f9 [zsxwing] Clean the codes\nb82ed1e [zsxwing] Update the graphs as per comments\ndd653a1 [zsxwing] Add timeline and histogram graphs for streaming statistics","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/489700c809a7c0a836538f3d0bd58bed609e8768"},{"sha":"735bc3d042b1e3e12de57b66f166af14254ad314","author":{"email":"c95969b69933e0f49af50a76ac2e9a9cf10c0740@alibaba-inc.com","name":"云峤"},"message":"[SPARK-7294][SQL] ADD BETWEEN\n\nAuthor: 云峤 <chensong.cs@alibaba-inc.com>\nAuthor: kaka1992 <kaka_1992@163.com>\n\nCloses #5839 from kaka1992/master and squashes the following commits:\n\nb15360d [kaka1992] Fix python unit test in sql/test. =_= I forget to commit this file last time.\nf928816 [kaka1992] Fix python style in sql/test.\nd2e7f72 [kaka1992] Fix python style in sql/test.\nc54d904 [kaka1992] Fix empty map bug.\n7e64d1e [云峤] Update\n7b9b858 [云峤] undo\nf080f8d [云峤] update pep8\n76f0c51 [云峤] Merge remote-tracking branch 'remotes/upstream/master'\n7d62368 [云峤] [SPARK-7294] ADD BETWEEN\nbaf839b [云峤] [SPARK-7294] ADD BETWEEN\nd11d5b9 [云峤] [SPARK-7294] ADD BETWEEN","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/735bc3d042b1e3e12de57b66f166af14254ad314"},{"sha":"fec7b29f5566b8a46b86f706c64c7823f95e0729","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-7351] [STREAMING] [DOCS] Add spark.streaming.ui.retainedBatches to docs\n\nThe default value will be changed to `1000` in #5533. So here I just used `1000`.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #5899 from zsxwing/SPARK-7351 and squashes the following commits:\n\ne1ec515 [zsxwing] [SPARK-7351][Streaming][Docs] Add spark.streaming.ui.retainedBatches to docs","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/fec7b29f5566b8a46b86f706c64c7823f95e0729"},{"sha":"3059291e2027fb3ebbb2d3376610ce450f7c15a3","author":{"email":"f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","name":"wangfei"},"message":"[SQL][Minor] make StringComparison extends ExpectsInputTypes\n\nmake StringComparison extends ExpectsInputTypes and added expectedChildTypes, so do not need override expectedChildTypes in each subclass\n\nAuthor: wangfei <wangfei1@huawei.com>\n\nCloses #5905 from scwf/ExpectsInputTypes and squashes the following commits:\n\nb374ddf [wangfei] make stringcomparison extends ExpectsInputTypes","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/3059291e2027fb3ebbb2d3376610ce450f7c15a3"},{"sha":"c688e3c5e46b26cb9fdba7987139c9ea63e2458b","author":{"email":"ad63b47dfac1951bde2b661ba9fbbe123f686bee@cs.berkeley.edu","name":"Shivaram Venkataraman"},"message":"[SPARK-7230] [SPARKR] Make RDD private in SparkR.\n\nThis change makes the RDD API private in SparkR and all internal uses of the SparkR API use SparkR::: to access private functions.\n\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\n\nCloses #5895 from shivaram/rrdd-private and squashes the following commits:\n\nbdb2f07 [Shivaram Venkataraman] Make RDD private in SparkR. This change also makes all internal uses of the SparkR API use SparkR::: to access private functions","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/c688e3c5e46b26cb9fdba7987139c9ea63e2458b"},{"sha":"0092abb47a0f9fdc716d5dfc1c591ddb45de8c98","author":{"email":"5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","name":"Sandy Ryza"},"message":"Some minor cleanup after SPARK-4550.\n\nJoshRosen this PR addresses the comments you left on #4450 after it got merged.\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #5916 from sryza/sandy-spark-4550-cleanup and squashes the following commits:\n\ndee3d85 [Sandy Ryza] Some minor cleanup after SPARK-4550.","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/0092abb47a0f9fdc716d5dfc1c591ddb45de8c98"},{"sha":"1fd31ba08928f8554f74609f48f4344bd69444e5","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-6231][SQL/DF] Automatically resolve join condition ambiguity for self-joins.\n\nSee the comment in join function for more information.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #5919 from rxin/self-join-resolve and squashes the following commits:\n\ne2fb0da [Reynold Xin] Updated SQLConf comment.\n7233a86 [Reynold Xin] Updated comment.\n6be2b4d [Reynold Xin] Removed println\n9f6b72f [Reynold Xin] [SPARK-6231][SQL/DF] Automatically resolve ambiguity in join condition for self-joins.","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/1fd31ba08928f8554f74609f48f4344bd69444e5"},{"sha":"51b3d41e160a1326a04536241b427e65b39ed8df","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Revert \"[SPARK-3454] separate json endpoints for data in the UI\"\n\nThis reverts commit d49735800db27239c11478aac4b0f2ec9df91a3f.\n\nThe commit broke Spark on Windows.","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/51b3d41e160a1326a04536241b427e65b39ed8df"},{"sha":"a4669443999dc13a1bb34509c827d8b9096ea84f","author":{"email":"3b63531f765b09216524bb8f92815227f77df3d2@intel.com","name":"qhuang"},"message":"[SPARK-6841] [SPARKR] add support for mean, median, stdev etc.\n\nMoving here from https://github.com/amplab-extras/SparkR-pkg/pull/241\nsum() has been implemented. (https://github.com/amplab-extras/SparkR-pkg/pull/242)\n\nNow Phase 1: mean, sd, var have been implemented, but some things still need to be improved with the suggestions in https://issues.apache.org/jira/browse/SPARK-6841\n\nAuthor: qhuang <qian.huang@intel.com>\n\nCloses #5446 from hqzizania/R and squashes the following commits:\n\nf283572 [qhuang] add test unit for describe()\n2e74d5a [qhuang] add describe() DataFrame API","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/a4669443999dc13a1bb34509c827d8b9096ea84f"},{"sha":"ba2b56614d7ab1bd7409b49e9d85c248d8faa48b","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":"[SPARK-7358][SQL] Move DataFrame mathfunctions into functions\n\nAfter a discussion on the user mailing list, it was decided to put all UDF's under `o.a.s.sql.functions`\n\ncc rxin\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #5923 from brkyvz/move-math-funcs and squashes the following commits:\n\na8dc3f7 [Burak Yavuz] address comments\ncf7a7bb [Burak Yavuz] [SPARK-7358] Move DataFrame mathfunctions into functions","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/ba2b56614d7ab1bd7409b49e9d85c248d8faa48b"},{"sha":"7b1457839bdac124a07fd6292f6263f0ded48880","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-6267] [MLLIB] Python API for IsotonicRegression\n\nhttps://issues.apache.org/jira/browse/SPARK-6267\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #5890 from yanboliang/spark-6267 and squashes the following commits:\n\nf20541d [Yanbo Liang] Merge pull request #3 from mengxr/SPARK-6267\n7f202f9 [Xiangrui Meng] use Vector to have the best Python 2&3 compatibility\n4bccfee [Yanbo Liang] fix doctest\nec09412 [Yanbo Liang] fix typos\n8214bbb [Yanbo Liang] fix code style\n5c8ebe5 [Yanbo Liang] Python API for IsotonicRegression","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/7b1457839bdac124a07fd6292f6263f0ded48880"},{"sha":"9f019c7223bb79b8d5cd52980b2723a1601d1134","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-7384][Core][Tests] Fix flaky tests for distributed mode in BroadcastSuite\n\nFixed the following failure: https://amplab.cs.berkeley.edu/jenkins/job/Spark-1.3-Maven-pre-YARN/hadoop.version=1.0.4,label=centos/452/testReport/junit/org.apache.spark.broadcast/BroadcastSuite/Unpersisting_HttpBroadcast_on_executors_and_driver_in_distributed_mode/\n\nThe tests should wait until all slaves are up. Otherwise, there may be only a part of `BlockManager`s registered, and fail the tests.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #5925 from zsxwing/SPARK-7384 and squashes the following commits:\n\n783cb7b [zsxwing] Add comments for _jobProgressListener and remove postfixOps\n1009ef1 [zsxwing] [SPARK-7384][Core][Tests] Fix flaky tests for distributed mode in BroadcastSuite","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/9f019c7223bb79b8d5cd52980b2723a1601d1134"},{"sha":"32cdc815c6fc19b5c8c4eca35f88a61302d67cd5","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-6940] [MLLIB] Add CrossValidator to Python ML pipeline API\n\nSince CrossValidator is a meta algorithm, we copy the implementation in Python. jkbradley\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #5926 from mengxr/SPARK-6940 and squashes the following commits:\n\n6af181f [Xiangrui Meng] add TODOs\n8285134 [Xiangrui Meng] update doc\n060f7c3 [Xiangrui Meng] update doctest\nacac727 [Xiangrui Meng] add keyword args\ncdddecd [Xiangrui Meng] add CrossValidator in Python","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/32cdc815c6fc19b5c8c4eca35f88a61302d67cd5"},{"sha":"322e7e7f689947aef29909572ee0c0e110ea23b8","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SQL] JavaDoc update for various DataFrame functions.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #5935 from rxin/df-doc1 and squashes the following commits:\n\naaeaadb [Reynold Xin] [SQL] JavaDoc update for various DataFrame functions.","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/322e7e7f689947aef29909572ee0c0e110ea23b8"},{"sha":"150f671c286c57deaf37ab1d8f837d68b5be82a0","author":{"email":"c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","name":"Daoyuan Wang"},"message":"[SPARK-5456] [SQL] fix decimal compare for jdbc rdd\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #5803 from adrian-wang/decimalcompare and squashes the following commits:\n\naef0e96 [Daoyuan Wang] add null handle\nec455b9 [Daoyuan Wang] fix decimal compare for jdbc rdd","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/150f671c286c57deaf37ab1d8f837d68b5be82a0"},{"sha":"c3eb441f5487c9b6476e1d6e2a2d852dcc43b986","author":{"email":"c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","name":"Daoyuan Wang"},"message":"[SPARK-6201] [SQL] promote string and do widen types for IN\n\nhuangjs\nAcutally spark sql will first go through analysis period, in which we do widen types and promote strings, and then optimization, where constant IN will be converted into INSET.\n\nSo it turn out that we only need to fix this for IN.\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #4945 from adrian-wang/inset and squashes the following commits:\n\n71e05cc [Daoyuan Wang] minor fix\n581fa1c [Daoyuan Wang] mysql way\nf3f7baf [Daoyuan Wang] address comments\n5eed4bc [Daoyuan Wang] promote string and do widen types for IN","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/c3eb441f5487c9b6476e1d6e2a2d852dcc43b986"},{"sha":"f2c47082c3412a4cf8cbabe12585147c5ec3ea40","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-1442] [SQL] Window Function Support for Spark SQL\n\nAdding more information about the implementation...\n\nThis PR is adding the support of window functions to Spark SQL (specifically OVER and WINDOW clause). For every expression having a OVER clause, we use a WindowExpression as the container of a WindowFunction and the corresponding WindowSpecDefinition (the definition of a window frame, i.e. partition specification, order specification, and frame specification appearing in a OVER clause).\n# Implementation #\nThe high level work flow of the implementation is described as follows.\n\n*\tQuery parsing: In the query parse process, all WindowExpressions are originally placed in the projectList of a Project operator or the aggregateExpressions of an Aggregate operator. It makes our changes to simple and keep all of parsing rules for window functions at a single place (nodesToWindowSpecification). For the WINDOWclause in a query, we use a WithWindowDefinition as the container as the mapping from the name of a window specification to a WindowSpecDefinition. This changes is similar with our common table expression support.\n\n*\tAnalysis: The query analysis process has three steps for window functions.\n\n *\tResolve all WindowSpecReferences by replacing them with WindowSpecReferences according to the mapping table stored in the node of WithWindowDefinition.\n *\tResolve WindowFunctions in the projectList of a Project operator or the aggregateExpressions of an Aggregate operator. For this PR, we use Hive's functions for window functions because we will have a major refactoring of our internal UDAFs and it is better to switch our UDAFs after that refactoring work.\n *\tOnce we have resolved all WindowFunctions, we will use ResolveWindowFunction to extract WindowExpressions from projectList and aggregateExpressions and then create a Window operator for every distinct WindowSpecDefinition. With this choice, at the execution time, we can rely on the Exchange operator to do all of work on reorganizing the table and we do not need to worry about it in the physical Window operator. An example analyzed plan is shown as follows\n\n```\nsql(\"\"\"\nSELECT\n  year, country, product, sales,\n  avg(sales) over(partition by product) avg_product,\n  sum(sales) over(partition by country) sum_country\nFROM sales\nORDER BY year, country, product\n\"\"\").explain(true)\n\n== Analyzed Logical Plan ==\nSort [year#34 ASC,country#35 ASC,product#36 ASC], true\n Project [year#34,country#35,product#36,sales#37,avg_product#27,sum_country#28]\n  Window [year#34,country#35,product#36,sales#37,avg_product#27], [HiveWindowFunction#org.apache.hadoop.hive.ql.udf.generic.GenericUDAFSum(sales#37) WindowSpecDefinition [country#35], [], ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING AS sum_country#28], WindowSpecDefinition [country#35], [], ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING\n   Window [year#34,country#35,product#36,sales#37], [HiveWindowFunction#org.apache.hadoop.hive.ql.udf.generic.GenericUDAFAverage(sales#37) WindowSpecDefinition [product#36], [], ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING AS avg_product#27], WindowSpecDefinition [product#36], [], ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING\n    Project [year#34,country#35,product#36,sales#37]\n     MetastoreRelation default, sales, None\n```\n\n*\tQuery planning: In the process of query planning, we simple generate the physical Window operator based on the logical Window operator. Then, to prepare the executedPlan, the EnsureRequirements rule will add Exchange and Sort operators if necessary. The EnsureRequirements rule will analyze the data properties and try to not add unnecessary shuffle and sort. The physical plan for the above example query is shown below.\n\n```\n== Physical Plan ==\nSort [year#34 ASC,country#35 ASC,product#36 ASC], true\n Exchange (RangePartitioning [year#34 ASC,country#35 ASC,product#36 ASC], 200), []\n  Window [year#34,country#35,product#36,sales#37,avg_product#27], [HiveWindowFunction#org.apache.hadoop.hive.ql.udf.generic.GenericUDAFSum(sales#37) WindowSpecDefinition [country#35], [], ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING AS sum_country#28], WindowSpecDefinition [country#35], [], ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING\n   Exchange (HashPartitioning [country#35], 200), [country#35 ASC]\n    Window [year#34,country#35,product#36,sales#37], [HiveWindowFunction#org.apache.hadoop.hive.ql.udf.generic.GenericUDAFAverage(sales#37) WindowSpecDefinition [product#36], [], ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING AS avg_product#27], WindowSpecDefinition [product#36], [], ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING\n     Exchange (HashPartitioning [product#36], 200), [product#36 ASC]\n      HiveTableScan [year#34,country#35,product#36,sales#37], (MetastoreRelation default, sales, None), None\n```\n\n*\tExecution time: At execution time, a physical Window operator buffers all rows in a partition specified in the partition spec of a OVER clause. If necessary, it also maintains a sliding window frame. The current implementation tries to buffer the input parameters of a window function according to the window frame to avoid evaluating a row multiple times.\n\n# Future work #\n\nHere are three improvements that are not hard to add:\n*\tTaking advantage of the window frame specification to reduce the number of rows buffered in the physical Window operator. For some cases, we only need to buffer the rows appearing in the sliding window. But for other cases, we will not be able to reduce the number of rows buffered (e.g. ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING).\n\n*\tWhen aRAGEN frame is used, for <value> PRECEDING and <value> FOLLOWING, it will be great if the <value> part is an expression (we can start with Literal). So, when the data type of ORDER BY expression is a FractionalType, we can support FractionalType as the type <value> (<value> still needs to be evaluated as a positive value).\n\n*\tWhen aRAGEN frame is used, we need to support DateType and TimestampType as the data type of the expression appearing in the order specification. Then, the <value> part of <value> PRECEDING and <value> FOLLOWING can support interval types (once we support them).\n\nThis is a joint work with guowei2 and yhuai\nThanks hbutani hvanhovell for his comments\nThanks scwf for his comments and unit tests\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #5604 from guowei2/windowImplement and squashes the following commits:\n\n76fe1c8 [Yin Huai] Implementation.\naa2b0ae [Yin Huai] Tests.","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/f2c47082c3412a4cf8cbabe12585147c5ec3ea40"},{"sha":"002c12384d6ecebbb3e7fc853dbdfbc5aaa3d6a6","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-7311] Introduce internal Serializer API for determining if serializers support object relocation\n\nThis patch extends the `Serializer` interface with a new `Private` API which allows serializers to indicate whether they support relocation of serialized objects in serializer stream output.\n\nThis relocatibilty property is described in more detail in `Serializer.scala`, but in a nutshell a serializer supports relocation if reordering the bytes of serialized objects in serialization stream output is equivalent to having re-ordered those elements prior to serializing them.  The optimized shuffle path introduced in #4450 and #5868 both rely on serializers having this property; this patch just centralizes the logic for determining whether a serializer has this property.  I also added tests and comments clarifying when this works for KryoSerializer.\n\nThis change allows the optimizations in #4450 to be applied for shuffles that use `SqlSerializer2`.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #5924 from JoshRosen/SPARK-7311 and squashes the following commits:\n\n50a68ca [Josh Rosen] Address minor nits\n0a7ebd7 [Josh Rosen] Clarify reason why SqlSerializer2 supports this serializer\n123b992 [Josh Rosen] Cleanup for submitting as standalone patch.\n4aa61b2 [Josh Rosen] Add missing newline\n2c1233a [Josh Rosen] Small refactoring of SerializerPropertiesSuite to enable test re-use:\n0ba75e6 [Josh Rosen] Add tests for serializer relocation property.\n450fa21 [Josh Rosen] Back out accidental log4j.properties change\n86d4dcd [Josh Rosen] Flag that SparkSqlSerializer2 supports relocation\nb9624ee [Josh Rosen] Expand serializer API and use new function to help control when new UnsafeShuffle path is used.","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/002c12384d6ecebbb3e7fc853dbdfbc5aaa3d6a6"},{"sha":"845d1d4d0cc5512d1671b2f2c7c74c5b97c3aef8","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"Add `Private` annotation.\n\nThis was originally added as part of #4435, which was reverted.","distinct":true,"url":"https://api.github.com/repos/aglahe/spark/commits/845d1d4d0cc5512d1671b2f2c7c74c5b97c3aef8"}]}