{"push_id":700423007,"size":67,"distinct_size":1,"ref":"refs/heads/type-check","head":"0faa15a8a206520edffb610ebfa449ba44c55a3a","before":"772866dfeccb446d499a1eef74ab44a820bd61a1","commits":[{"sha":"d46f8e5d4b5c1278e0fae3ad133b2229ac01b197","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-7186] [SQL] Decouple internal Row from external Row\n\nCurrently, we use o.a.s.sql.Row both internally and externally. The external interface is wider than what the internal needs because it is designed to facilitate end-user programming. This design has proven to be very error prone and cumbersome for internal Row implementations.\n\nAs a first step, we create an InternalRow interface in the catalyst module, which is identical to the current Row interface. And we switch all internal operators/expressions to use this InternalRow instead. When we need to expose Row, we convert the InternalRow implementation into Row for users.\n\nFor all public API, we use Row (for example, data source APIs), which will be converted into/from InternalRow by CatalystTypeConverters.\n\nFor all internal data sources (Json, Parquet, JDBC, Hive), we use InternalRow for better performance, casted into Row in buildScan() (without change the public API). When create a PhysicalRDD, we cast them back to InternalRow.\n\ncc rxin marmbrus JoshRosen\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #6792 from davies/internal_row and squashes the following commits:\n\nf2abd13 [Davies Liu] fix scalastyle\na7e025c [Davies Liu] move InternalRow into catalyst\n30db8ba [Davies Liu] Merge branch 'master' of github.com:apache/spark into internal_row\n7cbced8 [Davies Liu] separate Row and InternalRow","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/d46f8e5d4b5c1278e0fae3ad133b2229ac01b197"},{"sha":"4aed66f299a67f5a594da9316b6bf4c345838216","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-8329][SQL] Allow _ in DataSource options\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #6786 from marmbrus/optionsParser and squashes the following commits:\n\ne7d18ef [Michael Armbrust] add dots\n99a3452 [Michael Armbrust] [SPARK-8329][SQL] Allow _ in DataSource options","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/4aed66f299a67f5a594da9316b6bf4c345838216"},{"sha":"d986fb9a378416248768828e6e6c7405697f9a5a","author":{"email":"a6beb8a5d89069108e69bd55bd09a72bc68f527d@measite.de","name":"Rene Treffer"},"message":"[SPARK-7897] Improbe type for jdbc/\"unsigned bigint\"\n\nThe original fix uses DecimalType.Unlimited, which is harder to\nhandle afterwards. There is no scale and most data should fit into\na long, thus DecimalType(20,0) should be better.\n\nAuthor: Rene Treffer <treffer@measite.de>\n\nCloses #6789 from rtreffer/spark-7897-unsigned-bigint-as-decimal and squashes the following commits:\n\n2006613 [Rene Treffer] Fix type for \"unsigned bigint\" jdbc loading.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/d986fb9a378416248768828e6e6c7405697f9a5a"},{"sha":"ce1041c38f92449ca14894551c358c875672afe6","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-8346] [SQL] Use InternalRow instread of catalyst.InternalRow\n\ncc rxin marmbrus\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #6802 from davies/cleanup_internalrow and squashes the following commits:\n\n769d2aa [Davies Liu] remove not needed cast\n4acbbe4 [Davies Liu] catalyst.Internal -> InternalRow","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/ce1041c38f92449ca14894551c358c875672afe6"},{"sha":"af31335adce13e1452ce1990496c9bfac9778b5c","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-8319] [CORE] [SQL] Update logic related to key orderings in shuffle dependencies\n\nThis patch updates two pieces of logic that are related to handling of keyOrderings in ShuffleDependencies:\n\n- The Tungsten ShuffleManager falls back to regular SortShuffleManager whenever the shuffle dependency specifies a key ordering, but technically we only need to fall back when an aggregator is also specified. This patch updates the fallback logic to reflect this so that the Tungsten optimizations can apply to more workloads.\n\n- The SQL Exchange operator performs defensive copying of shuffle inputs when a key ordering is specified, but this is unnecessary. The copying was added to guard against cases where ExternalSorter would buffer non-serialized records in memory.  When ExternalSorter is configured without an aggregator, it uses the following logic to determine whether to buffer records in a serialized or deserialized format:\n\n   ```scala\n     private val useSerializedPairBuffer =\n        ordering.isEmpty &&\n        conf.getBoolean(\"spark.shuffle.sort.serializeMapOutputs\", true) &&\n        ser.supportsRelocationOfSerializedObjects\n   ```\n\n   The `newOrdering.isDefined` branch in `ExternalSorter.needToCopyObjectsBeforeShuffle`, removed by this patch, is not necessary:\n\n   - It was checked even if we weren't using sort-based shuffle, but this was unnecessary because only SortShuffleManager performs map-side sorting.\n   - Map-side sorting during shuffle writing is only performed for shuffles that perform map-side aggregation as part of the shuffle (to see this, look at how SortShuffleWriter constructs ExternalSorter).  Since SQL never pushes aggregation into Spark's shuffle, we can guarantee that both the aggregator and ordering will be empty and Spark SQL always uses serializers that support relocation, so sort-shuffle will use the serialized pair buffer unless the user has explicitly disabled it via the SparkConf feature-flag.  Therefore, I think my optimization in Exchange should be safe.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #6773 from JoshRosen/SPARK-8319 and squashes the following commits:\n\n7a14129 [Josh Rosen] Revise comments; add handler to guard against future ShuffleManager implementations\n07bb2c9 [Josh Rosen] Update comment to clarify circumstances under which shuffle operates on serialized records\n269089a [Josh Rosen] Avoid unnecessary copy in SQL Exchange\n34e526e [Josh Rosen] Enable Tungsten shuffle for non-agg shuffles w/ key orderings","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/af31335adce13e1452ce1990496c9bfac9778b5c"},{"sha":"ddec45279ed1061f4c05fd0760309a53581d03f5","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-8052] [SQL] Use java.math.BigDecimal for casting String to Decimal instead of using toDouble\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-8052\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #6645 from viirya/cast_string_integraltype and squashes the following commits:\n\ne19c6a3 [Liang-Chi Hsieh] For comment.\nc3e472a [Liang-Chi Hsieh] Add test.\n7ced9b0 [Liang-Chi Hsieh] Use java.math.BigDecimal for casting String to Decimal instead of using toDouble.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/ddec45279ed1061f4c05fd0760309a53581d03f5"},{"sha":"a138953391975886c88bfe81d4ce6b6dd189cd32","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-8347][SQL] Add unit tests for abs.\n\nAlso addressed code review feedback from #6754\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #6803 from rxin/abs and squashes the following commits:\n\nd07beba [Reynold Xin] [SPARK-8347] Add unit tests for abs.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/a138953391975886c88bfe81d4ce6b6dd189cd32"},{"sha":"2d71ba4c8a629deab672869ac8e8b6a4b3aec479","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-8349] [SQL] Use expression constructors (rather than apply) in FunctionRegistry\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #6806 from rxin/gs and squashes the following commits:\n\ned1aebb [Reynold Xin] Fixed style.\nc7fc3e6 [Reynold Xin] [SPARK-8349][SQL] Use expression constructors (rather than apply) in FunctionRegistry","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/2d71ba4c8a629deab672869ac8e8b6a4b3aec479"},{"sha":"35d1267cf8e918032c92a206b22bb301bf0c806e","author":{"email":"fcf9e35ad50aab44387f6ff709c4c4bf73ee4abe@gmail.com","name":"Mike Dusenberry"},"message":"[Spark-8343] [Streaming] [Docs] Improve Spark Streaming Guides.\n\nThis improves the Spark Streaming Guides by fixing broken links, rewording confusing sections, fixing typos, adding missing words, etc.\n\nAuthor: Mike Dusenberry <dusenberrymw@gmail.com>\n\nCloses #6801 from dusenberrymw/SPARK-8343_Improve_Spark_Streaming_Guides_MERGED and squashes the following commits:\n\n6688090 [Mike Dusenberry] Improvements to the Spark Streaming Custom Receiver Guide, including slight rewording of confusing sections, and fixing typos & missing words.\n436fbd8 [Mike Dusenberry] Bunch of improvements to the Spark Streaming Guide, including fixing broken links, slight rewording of confusing sections, fixing typos & missing words, etc.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/35d1267cf8e918032c92a206b22bb301bf0c806e"},{"sha":"cb7ada1196b5e13f0d4bb1988dc707a72003b8bc","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-8342][SQL] Fix Decimal setOrNull\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-8342\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #6797 from viirya/fix_decimal and squashes the following commits:\n\n8a447b1 [Liang-Chi Hsieh] Add unit test.\nd67a5ea [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into fix_decimal\nab6d8af [Liang-Chi Hsieh] Fix setOrNull.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/cb7ada1196b5e13f0d4bb1988dc707a72003b8bc"},{"sha":"ea7fd2ff6454e8d819a39bf49901074e49b5714e","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-8354] [SQL] Fix off-by-factor-of-8 error when allocating scratch space in UnsafeFixedWidthAggregationMap\n\nUnsafeFixedWidthAggregationMap contains an off-by-factor-of-8 error when allocating row conversion scratch space: we take a size requirement, measured in bytes, then allocate a long array of that size.  This means that we end up allocating 8x too much conversion space.\n\nThis patch fixes this by allocating a `byte[]` array instead.  This doesn't impose any new limitations on the maximum sizes of UnsafeRows, since UnsafeRowConverter already used integers when calculating the size requirements for rows.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #6809 from JoshRosen/sql-bytes-vs-words-fix and squashes the following commits:\n\n6520339 [Josh Rosen] Updates to reflect fact that UnsafeRow max size is constrained by max byte[] size","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/ea7fd2ff6454e8d819a39bf49901074e49b5714e"},{"sha":"9073a426e444e4bc6efa8608e54e0a986f38a270","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-8358] [SQL] Wait for child resolution when resolving generators\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #6811 from marmbrus/aliasExplodeStar and squashes the following commits:\n\nfbd2065 [Michael Armbrust] more style\n806a373 [Michael Armbrust] fix style\n7cbb530 [Michael Armbrust] [SPARK-8358][SQL] Wait for child resolution when resolving generatorsa","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/9073a426e444e4bc6efa8608e54e0a986f38a270"},{"sha":"53c16b92a537c392a7c3ebc3ef24c1e86cb1a7a4","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-8362] [SQL] Add unit tests for +, -, *, /, %\n\nAdded unit tests for all supported data types for:\n- Add\n- Subtract\n- Multiply\n- Divide\n- UnaryMinus\n- Remainder\n\nFixed bugs caught by the unit tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #6813 from rxin/SPARK-8362 and squashes the following commits:\n\nfb3fe62 [Reynold Xin] Added Remainder.\n3b266ba [Reynold Xin] [SPARK-8362] Add unit tests for +, -, *, /.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/53c16b92a537c392a7c3ebc3ef24c1e86cb1a7a4"},{"sha":"f3f2a4397da164f0ddfa5d60bf441099296c4346","author":{"email":"8df74dda184c19df19d4755602108540e071bf38@peter-hoffmann.com","name":"Peter Hoffmann"},"message":"fix read/write mixup\n\nAuthor: Peter Hoffmann <ph@peter-hoffmann.com>\n\nCloses #6815 from hoffmann/patch-1 and squashes the following commits:\n\n2abb6da [Peter Hoffmann] fix read/write mixup","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/f3f2a4397da164f0ddfa5d60bf441099296c4346"},{"sha":"4eb48ed1dadee80d78ada5d15884dd348c46ad27","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-8065] [SQL] Add support for Hive 0.14 metastores\n\nThis change has two parts.\n\nThe first one gets rid of \"ReflectionMagic\". That worked well for the differences between 0.12 and\n0.13, but breaks in 0.14, since some of the APIs that need to be used have primitive types. I could\nnot figure out a way to make that class work with primitive types. So instead I wrote some shims\n (I can already hear the collective sigh) that find the appropriate methods via reflection. This should\nbe faster since the method instances are cached, and the code is not much uglier than before,\nwith the advantage that all the ugliness is local to one file (instead of multiple switch statements on\nthe version being used scattered in ClientWrapper).\n\nThe second part is simple: add code to handle Hive 0.14. A few new methods had to be added\nto the new shims.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #6627 from vanzin/SPARK-8065 and squashes the following commits:\n\n3fa4270 [Marcelo Vanzin] Indentation style.\n4b8a3d4 [Marcelo Vanzin] Fix dep exclusion.\nbe3d0cc [Marcelo Vanzin] Merge branch 'master' into SPARK-8065\nca3fb1e [Marcelo Vanzin] Merge branch 'master' into SPARK-8065\nb43f13e [Marcelo Vanzin] Since exclusions seem to work, clean up some of the code.\n73bd161 [Marcelo Vanzin] Botched merge.\nd2ddf01 [Marcelo Vanzin] Comment about excluded dep.\n0c929d1 [Marcelo Vanzin] Merge branch 'master' into SPARK-8065\n2c3c02e [Marcelo Vanzin] Try to fix tests by adding support for exclusions.\n0a03470 [Marcelo Vanzin] Try to fix tests by upgrading calcite dependency.\n13b2dfa [Marcelo Vanzin] Fix NPE.\n6439d88 [Marcelo Vanzin] Minor style thing.\n69b017b [Marcelo Vanzin] Style.\na21cad8 [Marcelo Vanzin] Part II: Add shims / version for Hive 0.14.\nae98c87 [Marcelo Vanzin] PART I: Get rid of reflection magic.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/4eb48ed1dadee80d78ada5d15884dd348c46ad27"},{"sha":"4c5889e8f5fd384a3a33e49d75a64cde95d2c9f3","author":{"email":"fbfd45890515f7eeeeabf87d7a80d3449c3bce83@gmail.com","name":"Nicholas Chammas"},"message":"[SPARK-8316] Upgrade to Maven 3.3.3\n\nVersions of Maven older than 3.3.0 apparently have [a bug in how they handle transitive dependencies](https://github.com/apache/spark/pull/6492#issuecomment-111001101).\n\nI confirmed that upgrading to Maven 3.3.3 resolves at least the particular manifestation of this bug that I ran into.\n\nAuthor: Nicholas Chammas <nicholas.chammas@gmail.com>\n\nCloses #6770 from nchammas/maven-333 and squashes the following commits:\n\n6bed2d9 [Nicholas Chammas] upgrade to Maven 3.3.3","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/4c5889e8f5fd384a3a33e49d75a64cde95d2c9f3"},{"sha":"56d4e8a2d0f6aab9a599cd8733e20500ffe8fc8a","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"andrewor14"},"message":"[SPARK-8350] [R] Log R unit test output to \"unit-tests.log\"\n\nRight now it's logged to \"R-unit-tests.log\". Jenkins currently only archives files named \"unit-tests.log\", and this is what all other modules (e.g. SQL, network, REPL) use.\n1. We should be consistent\n2. I don't want to reconfigure Jenkins to accept a different file\n\nshivaram\n\nAuthor: andrewor14 <andrew@databricks.com>\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #6807 from andrewor14/r-logs and squashes the following commits:\n\n96005d2 [andrewor14] Nest unit-tests.log further until R\n407c46c [andrewor14] Add target to log path\nd7b68ae [Andrew Or] Log R unit test output to \"unit-tests.log\"","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/56d4e8a2d0f6aab9a599cd8733e20500ffe8fc8a"},{"sha":"6ae21a944a0f4580b55749776223c827450b00da","author":{"email":"88ec0e7df70fbb4dfef58da07dfeb1a4a708d199@gmail.com","name":"Yadong Qi"},"message":"[SPARK-6583] [SQL] Support aggregate functions in ORDER BY\n\nAdd aggregates in ORDER BY clauses to the `Aggregate` operator beneath.  Project these results away after the Sort.\n\nBased on work by watermen.  Also Closes #5290.\n\nAuthor: Yadong Qi <qiyadong2010@gmail.com>\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #6816 from marmbrus/pr/5290 and squashes the following commits:\n\n3226a97 [Michael Armbrust] consistent ordering\neb8938d [Michael Armbrust] no vars\nc8b25c1 [Yadong Qi] move the test data.\n7f9b736 [Yadong Qi] delete Substring case\na1e87c1 [Yadong Qi] fix conflict\nf119849 [Yadong Qi] order by aggregated function","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/6ae21a944a0f4580b55749776223c827450b00da"},{"sha":"1a62d61696a0481508d83a07d19ab3701245ac20","author":{"email":"22f3e725b314afeca4d45e12683c560674029d21@gmail.com","name":"tedyu"},"message":"SPARK-8336 Fix NullPointerException with functions.rand()\n\nThis PR fixes the problem reported by Justin Yip in the thread 'NullPointerException with functions.rand()'\n\nTested using spark-shell and verified that the following works:\nsqlContext.createDataFrame(Seq((1,2), (3, 100))).withColumn(\"index\", rand(30)).show()\n\nAuthor: tedyu <yuzhihong@gmail.com>\n\nCloses #6793 from tedyu/master and squashes the following commits:\n\n62fd97b [tedyu] Create RandomSuite\n750f92c [tedyu] Add test for Rand() with seed\na1d66c5 [tedyu] Fix NullPointerException with functions.rand()","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/1a62d61696a0481508d83a07d19ab3701245ac20"},{"sha":"bc76a0f7506c9796209a96b027a236270c23bbf6","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-7184] [SQL] enable codegen by default\n\nIn order to have better performance out of box, this PR turn on codegen by default, then codegen can be tested by sql/test and hive/test.\n\nThis PR also fix some corner cases for codegen.\n\nBefore 1.5 release, we should re-visit this, turn it off if it's not stable or causing regressions.\n\ncc rxin JoshRosen\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #6726 from davies/enable_codegen and squashes the following commits:\n\nf3b25a5 [Davies Liu] fix warning\n73750ea [Davies Liu] fix long overflow when compare\n3017a47 [Davies Liu] Merge branch 'master' of github.com:apache/spark into enable_codegen\na7d75da [Davies Liu] Merge branch 'master' of github.com:apache/spark into enable_codegen\nff5b75a [Davies Liu] Merge branch 'master' of github.com:apache/spark into enable_codegen\nf4cf2c2 [Davies Liu] fix style\n99fc139 [Davies Liu] Merge branch 'enable_codegen' of github.com:davies/spark into enable_codegen\n91fc7a2 [Davies Liu] disable codegen for ScalaUDF\n207e339 [Davies Liu] Update CodeGenerator.scala\n44573a3 [Davies Liu] check thread safety of expression\nf3886fa [Davies Liu] don't inline primitiveTerm for null literal\nc8e7cd2 [Davies Liu] address comment\na8618c9 [Davies Liu] enable codegen by default","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/bc76a0f7506c9796209a96b027a236270c23bbf6"}]}