{"push_id":753524350,"size":34,"distinct_size":34,"ref":"refs/heads/sdev-udaf","head":"3c117342b40dd1fdd9de98c6a5cec94f5accb376","before":"27ae62566e1f34a407ecde7fb8fef0415073c466","commits":[{"sha":"76eaa701833a2ff23b50147d70ced41e85719572","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9674][SPARK-9667] Remove SparkSqlSerializer2\n\nIt is now subsumed by various Tungsten operators.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #7981 from rxin/SPARK-9674 and squashes the following commits:\n\n144f96e [Reynold Xin] Re-enable test\n58b7332 [Reynold Xin] Disable failing list.\nfb797e3 [Reynold Xin] Match all UDTs.\nbe9f243 [Reynold Xin] Updated if.\n71fc99c [Reynold Xin] [SPARK-9674][SPARK-9667] Remove GeneratedAggregate & SparkSqlSerializer2.","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/76eaa701833a2ff23b50147d70ced41e85719572"},{"sha":"2432c2e239f66049a7a7d7e0591204abcc993f1a","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-8382] [SQL] Improve Analysis Unit test framework\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #8025 from cloud-fan/analysis and squashes the following commits:\n\n51461b1 [Wenchen Fan] move test file to test folder\nec88ace [Wenchen Fan] Improve Analysis Unit test framework","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/2432c2e239f66049a7a7d7e0591204abcc993f1a"},{"sha":"9897cc5e3d6c70f7e45e887e2c6fc24dfa1adada","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9736] [SQL] JoinedRow.anyNull should delegate to the underlying rows.\n\nJoinedRow.anyNull currently loops through every field to check for null, which is inefficient if the underlying rows are UnsafeRows. It should just delegate to the underlying implementation.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #8027 from rxin/SPARK-9736 and squashes the following commits:\n\n03a2e92 [Reynold Xin] Include all files.\n90f1add [Reynold Xin] [SPARK-9736][SQL] JoinedRow.anyNull should delegate to the underlying rows.","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/9897cc5e3d6c70f7e45e887e2c6fc24dfa1adada"},{"sha":"aeddeafc03d77a5149d2c8f9489b0ca83e6b3e03","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9667][SQL] followup: Use GenerateUnsafeProjection.canSupport to test Exchange supported data types.\n\nThis way we recursively test the data types.\n\ncc chenghao-intel\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #8036 from rxin/cansupport and squashes the following commits:\n\nf7302ff [Reynold Xin] Can GenerateUnsafeProjection.canSupport to test Exchange supported data types.","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/aeddeafc03d77a5149d2c8f9489b0ca83e6b3e03"},{"sha":"05d04e10a8ea030bea840c3c5ba93ecac479a039","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9733][SQL] Improve physical plan explain for data sources\n\nAll data sources show up as \"PhysicalRDD\" in physical plan explain. It'd be better if we can show the name of the data source.\n\nWithout this patch:\n```\n== Physical Plan ==\nNewAggregate with UnsafeHybridAggregationIterator ArrayBuffer(date#0, cat#1) ArrayBuffer((sum(CAST((CAST(count#2, IntegerType) + 1), LongType))2,mode=Final,isDistinct=false))\n Exchange hashpartitioning(date#0,cat#1)\n  NewAggregate with UnsafeHybridAggregationIterator ArrayBuffer(date#0, cat#1) ArrayBuffer((sum(CAST((CAST(count#2, IntegerType) + 1), LongType))2,mode=Partial,isDistinct=false))\n   PhysicalRDD [date#0,cat#1,count#2], MapPartitionsRDD[3] at\n```\n\nWith this patch:\n```\n== Physical Plan ==\nTungstenAggregate(key=[date#0,cat#1], value=[(sum(CAST((CAST(count#2, IntegerType) + 1), LongType)),mode=Final,isDistinct=false)]\n Exchange hashpartitioning(date#0,cat#1)\n  TungstenAggregate(key=[date#0,cat#1], value=[(sum(CAST((CAST(count#2, IntegerType) + 1), LongType)),mode=Partial,isDistinct=false)]\n   ConvertToUnsafe\n    Scan ParquetRelation[file:/scratch/rxin/spark/sales4][date#0,cat#1,count#2]\n```\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #8024 from rxin/SPARK-9733 and squashes the following commits:\n\n811b90e [Reynold Xin] Fixed Python test case.\n52cab77 [Reynold Xin] Cast.\neea9ccc [Reynold Xin] Fix test case.\nfcecb22 [Reynold Xin] [SPARK-9733][SQL] Improve explain message for data source scan node.","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/05d04e10a8ea030bea840c3c5ba93ecac479a039"},{"sha":"881548ab20fa4c4b635c51d956b14bd13981e2f4","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-9674] Re-enable ignored test in SQLQuerySuite\n\nThe original code that this test tests is removed in https://github.com/apache/spark/commit/9270bd06fd0b16892e3f37213b5bc7813ea11fdd. It was ignored shortly before that so we never caught it. This patch re-enables the test and adds the code necessary to make it pass.\n\nJoshRosen yhuai\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #8015 from andrewor14/SPARK-9674 and squashes the following commits:\n\n225eac2 [Andrew Or] Merge branch 'master' of github.com:apache/spark into SPARK-9674\n8c24209 [Andrew Or] Fix NPE\ne541d64 [Andrew Or] Track aggregation memory for both sort and hash\n0be3a42 [Andrew Or] Fix test","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/881548ab20fa4c4b635c51d956b14bd13981e2f4"},{"sha":"e2fbbe73111d4624390f596a19a1799c86a05f6c","author":{"email":"3c24bdb07f4a2da8cf8b99806f22774158b31658@gmail.com","name":"Dariusz Kobylarz"},"message":"[SPARK-8481] [MLLIB] GaussianMixtureModel predict accepting single vector\n\nResubmit of [https://github.com/apache/spark/pull/6906] for adding single-vec predict to GMMs\n\nCC: dkobylarz  mengxr\n\nTo be merged with master and branch-1.5\nPrimary author: dkobylarz\n\nAuthor: Dariusz Kobylarz <darek.kobylarz@gmail.com>\n\nCloses #8039 from jkbradley/gmm-predict-vec and squashes the following commits:\n\nbfbedc4 [Dariusz Kobylarz] [SPARK-8481] [MLlib] GaussianMixtureModel predict accepting single vector","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/e2fbbe73111d4624390f596a19a1799c86a05f6c"},{"sha":"902334fd55bbe40a57c1de2a9bdb25eddf1c8cf6","author":{"email":"1e53f684e3f83d9ec103dadf5850d9f20f620973@users.noreply.github.com","name":"Bertrand Dechoux"},"message":"[SPARK-9748] [MLLIB] Centriod typo in KMeansModel\n\nA minor typo (centriod -> centroid). Readable variable names help every users.\n\nAuthor: Bertrand Dechoux <BertrandDechoux@users.noreply.github.com>\n\nCloses #8037 from BertrandDechoux/kmeans-typo and squashes the following commits:\n\n47632fe [Bertrand Dechoux] centriod typo","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/902334fd55bbe40a57c1de2a9bdb25eddf1c8cf6"},{"sha":"49702bd738de681255a7177339510e0e1b25a8db","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-8890] [SQL] Fallback on sorting when writing many dynamic partitions\n\nPreviously, we would open a new file for each new dynamic written out using `HadoopFsRelation`.  For formats like parquet this is very costly due to the buffers required to get good compression.  In this PR I refactor the code allowing us to fall back on an external sort when many partitions are seen.  As such each task will open no more than `spark.sql.sources.maxFiles` files.  I also did the following cleanup:\n\n - Instead of keying the file HashMap on an expensive to compute string representation of the partition, we now use a fairly cheap UnsafeProjection that avoids heap allocations.\n - The control flow for instantiating and invoking a writer container has been simplified.  Now instead of switching in two places based on the use of partitioning, the specific writer container must implement a single method `writeRows` that is invoked using `runJob`.\n - `InternalOutputWriter` has been removed.  Instead we have a `private[sql]` method `writeInternal` that converts and calls the public method.  This method can be overridden by internal datasources to avoid the conversion.  This change remove a lot of code duplication and per-row `asInstanceOf` checks.\n - `commands.scala` has been split up.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #8010 from marmbrus/fsWriting and squashes the following commits:\n\n00804fe [Michael Armbrust] use shuffleMemoryManager.pageSizeBytes\n775cc49 [Michael Armbrust] Merge remote-tracking branch 'origin/master' into fsWriting\n17b690e [Michael Armbrust] remove comment\n40f0372 [Michael Armbrust] address comments\nf5675bd [Michael Armbrust] char -> string\n7e2d0a4 [Michael Armbrust] make sure we close current writer\n8100100 [Michael Armbrust] delete empty commands.scala\n71cc717 [Michael Armbrust] update comment\n8ec75ac [Michael Armbrust] [SPARK-8890][SQL] Fallback on sorting when writing many dynamic partitions","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/49702bd738de681255a7177339510e0e1b25a8db"},{"sha":"cd540c1e59561ad1fdac59af6170944c60e685d8","author":{"email":"9835e449f9a666ddd8ba8aeb460af39a5130691b@databricks.com","name":"Feynman Liang"},"message":"[SPARK-9756] [ML] Make constructors in ML decision trees private\n\nThese should be made private until there is a public constructor for providing `rootNode: Node` to use these constructors.\n\njkbradley\n\nAuthor: Feynman Liang <fliang@databricks.com>\n\nCloses #8046 from feynmanliang/SPARK-9756 and squashes the following commits:\n\n2cbdf08 [Feynman Liang] Make RFRegressionModel aux constructor private\na06f596 [Feynman Liang] Make constructors in ML decision trees private","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/cd540c1e59561ad1fdac59af6170944c60e685d8"},{"sha":"85be65b39ce669f937a898195a844844d757666b","author":{"email":"9835e449f9a666ddd8ba8aeb460af39a5130691b@databricks.com","name":"Feynman Liang"},"message":"[SPARK-9719] [ML] Clean up Naive Bayes doc\n\nSmall documentation cleanups, including:\n * Adds documentation for `pi` and `theta`\n * setParam to `setModelType`\n\nAuthor: Feynman Liang <fliang@databricks.com>\n\nCloses #8047 from feynmanliang/SPARK-9719 and squashes the following commits:\n\nb372438 [Feynman Liang] Clean up naive bayes doc","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/85be65b39ce669f937a898195a844844d757666b"},{"sha":"998f4ff94df1d9db1c9e32c04091017c25cd4e81","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9754][SQL] Remove TypeCheck in debug package.\n\nTypeCheck no longer applies in the new \"Tungsten\" world.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #8043 from rxin/SPARK-9754 and squashes the following commits:\n\n4ec471e [Reynold Xin] [SPARK-9754][SQL] Remove TypeCheck in debug package.","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/998f4ff94df1d9db1c9e32c04091017c25cd4e81"},{"sha":"c564b27447ed99e55b359b3df1d586d5766b85ea","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-9753] [SQL] TungstenAggregate should also accept InternalRow instead of just UnsafeRow\n\nhttps://issues.apache.org/jira/browse/SPARK-9753\n\nThis PR makes TungstenAggregate to accept `InternalRow` instead of just `UnsafeRow`. Also, it adds an `getAggregationBufferFromUnsafeRow` method to `UnsafeFixedWidthAggregationMap`. It is useful when we already have grouping keys stored in `UnsafeRow`s. Finally, it wraps `InputStream` and `OutputStream` in `UnsafeRowSerializer` with `BufferedInputStream` and `BufferedOutputStream`, respectively.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #8041 from yhuai/joinedRowForProjection and squashes the following commits:\n\n7753e34 [Yin Huai] Use BufferedInputStream and BufferedOutputStream.\nd68b74e [Yin Huai] Use joinedRow instead of UnsafeRowJoiner.\ne93c009 [Yin Huai] Add getAggregationBufferFromUnsafeRow for cases that the given groupingKeyRow is already an UnsafeRow.","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/c564b27447ed99e55b359b3df1d586d5766b85ea"},{"sha":"ef062c15992b0d08554495b8ea837bef3fabf6e9","author":{"email":"dbe0629cf0118f7dffec38856cad0ec7ac880251@intel.com","name":"Carson Wang"},"message":"[SPARK-9731] Standalone scheduling incorrect cores if spark.executor.cores is not set\n\nThe issue only happens if `spark.executor.cores` is not set and executor memory is set to a high value.\nFor example, if we have a worker with 4G and 10 cores and we set `spark.executor.memory` to 3G, then only 1 core is assigned to the executor. The correct number should be 10 cores.\nI've added a unit test to illustrate the issue.\n\nAuthor: Carson Wang <carson.wang@intel.com>\n\nCloses #8017 from carsonwang/SPARK-9731 and squashes the following commits:\n\nd09ec48 [Carson Wang] Fix code style\n86b651f [Carson Wang] Simplify the code\n943cc4c [Carson Wang] fix scheduling correct cores to executors","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/ef062c15992b0d08554495b8ea837bef3fabf6e9"},{"sha":"11caf1ce290b6931647c2f71268f847d1d48930e","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-4176] [SQL] [MINOR] Should use unscaled Long to write decimals for precision <= 18 rather than 8\n\nThis PR fixes a minor bug introduced in #7455: when writing decimals, we should use the unscaled Long for better performance when the precision <= 18 rather than 8 (should be a typo). This bug doesn't affect correctness, but hurts Parquet decimal writing performance.\n\nThis PR also replaced similar magic numbers with newly defined constants.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #8031 from liancheng/spark-4176/minor-fix-for-writing-decimals and squashes the following commits:\n\n10d4ea3 [Cheng Lian] Should use unscaled Long to write decimals for precision <= 18 rather than 8","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/11caf1ce290b6931647c2f71268f847d1d48930e"},{"sha":"106c0789d8c83c7081bc9a335df78ba728e95872","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-9738] [SQL] remove FromUnsafe and add its codegen version to GenerateSafe\n\nIn https://github.com/apache/spark/pull/7752 we added `FromUnsafe` to convert nexted unsafe data like array/map/struct to safe versions. It's a quick solution and we already have `GenerateSafe` to do the conversion which is codegened. So we should remove `FromUnsafe` and implement its codegen version in `GenerateSafe`.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #8029 from cloud-fan/from-unsafe and squashes the following commits:\n\ned40d8f [Wenchen Fan] add the copy back\na93fd4b [Wenchen Fan] cogengen FromUnsafe","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/106c0789d8c83c7081bc9a335df78ba728e95872"},{"sha":"74a6541aa82bcd7a052b2e57b5ca55b7c316495b","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-4561] [PYSPARK] [SQL] turn Row into dict recursively\n\nAdd an option `recursive` to `Row.asDict()`, when True (default is False), it will convert the nested Row into dict.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #8006 from davies/as_dict and squashes the following commits:\n\n922cc5a [Davies Liu] turn Row into dict recursively","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/74a6541aa82bcd7a052b2e57b5ca55b7c316495b"},{"sha":"ac507a03c3371cd5404ca195ee0ba0306badfc23","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-6902] [SQL] [PYSPARK] Row should be read-only\n\nRaise an read-only exception when user try to mutable a Row.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #8009 from davies/readonly_row and squashes the following commits:\n\n8722f3f [Davies Liu] add tests\n05a3d36 [Davies Liu] Row should be read-only","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/ac507a03c3371cd5404ca195ee0ba0306badfc23"},{"sha":"23695f1d2d7ef9f3ea92cebcd96b1cf0e8904eb4","author":{"email":"348ff379544b45aa3f16e53a543620141423a0f6@gmail.com","name":"Yijie Shen"},"message":"[SPARK-9728][SQL]Support CalendarIntervalType in HiveQL\n\nThis PR enables converting interval term in HiveQL to CalendarInterval Literal.\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-9728\n\nAuthor: Yijie Shen <henry.yijieshen@gmail.com>\n\nCloses #8034 from yjshen/interval_hiveql and squashes the following commits:\n\n7fe9a5e [Yijie Shen] declare throw exception and add unit test\nfce7795 [Yijie Shen] convert hiveql interval term into CalendarInterval literal","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/23695f1d2d7ef9f3ea92cebcd96b1cf0e8904eb4"},{"sha":"a3aec918bed22f8e33cf91dc0d6e712e6653c7d2","author":{"email":"fe0966a084bd02ed13ddc43bf288232282ff6a11@cloudera.com","name":"Joseph Batchik"},"message":"[SPARK-9486][SQL] Add data source aliasing for external packages\n\nUsers currently have to provide the full class name for external data sources, like:\n\n`sqlContext.read.format(\"com.databricks.spark.avro\").load(path)`\n\nThis allows external data source packages to register themselves using a Service Loader so that they can add custom alias like:\n\n`sqlContext.read.format(\"avro\").load(path)`\n\nThis makes it so that using external data source packages uses the same format as the internal data sources like parquet, json, etc.\n\nAuthor: Joseph Batchik <joseph.batchik@cloudera.com>\nAuthor: Joseph Batchik <josephbatchik@gmail.com>\n\nCloses #7802 from JDrit/service_loader and squashes the following commits:\n\n49a01ec [Joseph Batchik] fixed a couple of format / error bugs\ne5e93b2 [Joseph Batchik] modified rat file to only excluded added services\n72b349a [Joseph Batchik] fixed error with orc data source actually\n9f93ea7 [Joseph Batchik] fixed error with orc data source\n87b7f1c [Joseph Batchik] fixed typo\n101cd22 [Joseph Batchik] removing unneeded changes\n8f3cf43 [Joseph Batchik] merged in changes\nb63d337 [Joseph Batchik] merged in master\n95ae030 [Joseph Batchik] changed the new trait to be used as a mixin for data source to register themselves\n74db85e [Joseph Batchik] reformatted class loader\nac2270d [Joseph Batchik] removing some added test\na6926db [Joseph Batchik] added test cases for data source loader\n208a2a8 [Joseph Batchik] changes to do error catching if there are multiple data sources\n946186e [Joseph Batchik] started working on service loader","distinct":true,"url":"https://api.github.com/repos/brkyvz/spark/commits/a3aec918bed22f8e33cf91dc0d6e712e6653c7d2"}]}