{"push_id":718852964,"size":43,"distinct_size":43,"ref":"refs/heads/master","head":"4ca90935c5ff15ccca13452572e317b86814f238","before":"9fb6b832bcc2556aa9db2981106cbd09f2959031","commits":[{"sha":"f32487b7ca86f768336a7c9b173f7c610fcde86f","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-8777] [SQL] Add random data generator test utilities to Spark SQL\n\nThis commit adds a set of random data generation utilities to Spark SQL, for use in its own unit tests.\n\n- `RandomDataGenerator.forType(DataType)` returns an `Option[() => Any]` that, if defined, contains a function for generating random values for the given DataType.  The random values use the external representations for the given DataType (for example, for DateType we return `java.sql.Date` instances instead of longs).\n- `DateTypeTestUtilities` defines some convenience fields for looping over instances of data types.  For example, `numericTypes` holds `DataType` instances for all supported numeric types.  These constants will help us to raise the level of abstraction in our tests.  For example, it's now very easy to write a test which is parameterized by all common data types.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #7176 from JoshRosen/sql-random-data-generators and squashes the following commits:\n\nf71634d [Josh Rosen] Roll back ScalaCheck usage\ne0d7d49 [Josh Rosen] Bump ScalaCheck version in LICENSE\n89d86b1 [Josh Rosen] Bump ScalaCheck version.\n0c20905 [Josh Rosen] Initial attempt at using ScalaCheck.\nb55875a [Josh Rosen] Generate doubles and floats over entire possible range.\n5acdd5c [Josh Rosen] Infinity and NaN are interesting.\nab76cbd [Josh Rosen] Move code to Catalyst package.\nd2b4a4a [Josh Rosen] Add random data generator test utilities to Spark SQL.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/f32487b7ca86f768336a7c9b173f7c610fcde86f"},{"sha":"f35b0c3436898f22860d2c6c1d12f3a661005201","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-8238][SPARK-8239][SPARK-8242][SPARK-8243][SPARK-8268][SQL]Add ascii/base64/unbase64/encode/decode functions\n\nAdd `ascii`,`base64`,`unbase64`,`encode` and `decode` expressions.\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #6843 from chenghao-intel/str_funcs2 and squashes the following commits:\n\n78dee7d [Cheng Hao] base 64 -> base64\n9d6f9f4 [Cheng Hao] remove the toString method for expressions\ned5c19c [Cheng Hao] update code as comments\n96170fc [Cheng Hao] scalastyle issues\ne2df768 [Cheng Hao] remove the unused import\n491ce7b [Cheng Hao] add ascii/base64/unbase64/encode/decode functions","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/f35b0c3436898f22860d2c6c1d12f3a661005201"},{"sha":"6b3574e68704d58ba41efe0ea4fe928cc166afcd","author":{"email":"8a087ed90f1ccc6f5e66d77d8aeba021c1ed975c@googlemail.com","name":"Tarek Auel"},"message":"[SPARK-8270][SQL] levenshtein distance\n\nJira: https://issues.apache.org/jira/browse/SPARK-8270\n\nInfo: I can not build the latest master, it stucks during the build process: `[INFO] Dependency-reduced POM written at: /Users/tarek/test/spark/bagel/dependency-reduced-pom.xml`\n\nAuthor: Tarek Auel <tarek.auel@googlemail.com>\n\nCloses #7214 from tarekauel/SPARK-8270 and squashes the following commits:\n\nab348b9 [Tarek Auel] Merge branch 'master' into SPARK-8270\na2ad318 [Tarek Auel] [SPARK-8270] changed order of fields\nd91b12c [Tarek Auel] [SPARK-8270] python fix\nadbd075 [Tarek Auel] [SPARK-8270] fixed typo\n23185c9 [Tarek Auel] [SPARK-8270] levenshtein distance","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/6b3574e68704d58ba41efe0ea4fe928cc166afcd"},{"sha":"48f7aed686afde70a6f0802c6cb37b0cad0509f1","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Fixed minor style issue with the previous merge.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/48f7aed686afde70a6f0802c6cb37b0cad0509f1"},{"sha":"347cab85cd924ffd326f3d1367b3b156ee08052d","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SQL] More unit tests for implicit type cast & add simpleString to AbstractDataType.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #7221 from rxin/implicit-cast-tests and squashes the following commits:\n\n64b13bd [Reynold Xin] Fixed a bug ..\n489b732 [Reynold Xin] [SQL] More unit tests for implicit type cast & add simpleString to AbstractDataType.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/347cab85cd924ffd326f3d1367b3b156ee08052d"},{"sha":"c991ef5abbb501933b2a68eea1987cf8d88794a5","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-8822][SQL] clean up type checking in math.scala.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #7220 from rxin/SPARK-8822 and squashes the following commits:\n\n0cda076 [Reynold Xin] Test cases.\n22d0463 [Reynold Xin] Fixed type precedence.\nbeb2a97 [Reynold Xin] [SPARK-8822][SQL] clean up type checking in math.scala.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/c991ef5abbb501933b2a68eea1987cf8d88794a5"},{"sha":"2b820f2a4bf9b154762e7516a5b0485322799da9","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[MINOR] [SQL] Minor fix for CatalystSchemaConverter\n\nping liancheng\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #7224 from viirya/few_fix_catalystschema and squashes the following commits:\n\nd994330 [Liang-Chi Hsieh] Minor fix for CatalystSchemaConverter.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/2b820f2a4bf9b154762e7516a5b0485322799da9"},{"sha":"f9c448dce8139e85ac564daa0f7e0325e778cffe","author":{"email":"957baac97c26135d676a6ed5e6e0814b0658c8e8@gmail.com","name":"Joshi"},"message":"[SPARK-7137] [ML] Update SchemaUtils checkInputColumn to print more info if needed\n\nAuthor: Joshi <rekhajoshm@gmail.com>\nAuthor: Rekha Joshi <rekhajoshm@gmail.com>\n\nCloses #5992 from rekhajoshm/fix/SPARK-7137 and squashes the following commits:\n\n8c42b57 [Joshi] update checkInputColumn to print more info if needed\n33ddd2e [Joshi] update checkInputColumn to print more info if needed\nacf3e17 [Joshi] update checkInputColumn to print more info if needed\n8993c0e [Joshi] SPARK-7137: Add checkInputColumn back to Params and print more info\ne3677c9 [Rekha Joshi] Merge pull request #1 from apache/master","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/f9c448dce8139e85ac564daa0f7e0325e778cffe"},{"sha":"a0cb111b22cb093e86b0daeecb3dcc41d095df40","author":{"email":"bc4a65f227ddcd81bb538c67e810a3e11696e8bf@gmail.com","name":"Yu ISHIKAWA"},"message":"[SPARK-8549] [SPARKR] Fix the line length of SparkR\n\n[[SPARK-8549] Fix the line length of SparkR - ASF JIRA](https://issues.apache.org/jira/browse/SPARK-8549)\n\nAuthor: Yu ISHIKAWA <yuu.ishikawa@gmail.com>\n\nCloses #7204 from yu-iskw/SPARK-8549 and squashes the following commits:\n\n6fb131a [Yu ISHIKAWA] Fix the typo\n1737598 [Yu ISHIKAWA] [SPARK-8549][SparkR] Fix the line length of SparkR","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/a0cb111b22cb093e86b0daeecb3dcc41d095df40"},{"sha":"6d0411b4f3a202cfb53f638ee5fd49072b42d3a6","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SQL][Minor] Update the DataFrame API for encode/decode\n\nThis is a the follow up of #6843.\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #7230 from chenghao-intel/str_funcs2_followup and squashes the following commits:\n\n52cc553 [Cheng Hao] update the code as comment","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/6d0411b4f3a202cfb53f638ee5fd49072b42d3a6"},{"sha":"86768b7b3b0c2964e744bc491bc20a1d3140ce93","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-8831][SQL] Support AbstractDataType in TypeCollection.\n\nOtherwise it is impossible to declare an expression supporting DecimalType.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #7232 from rxin/typecollection-adt and squashes the following commits:\n\n934d3d1 [Reynold Xin] [SPARK-8831][SQL] Support AbstractDataType in TypeCollection.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/86768b7b3b0c2964e744bc491bc20a1d3140ce93"},{"sha":"39e4e7e4d89077a637c4cad3a986e0e3447d1ae7","author":{"email":"6e84d8fe520cfe596aa55d1ec8fa60f90e47d128@engineersgatelp.com","name":"Steve Lindemann"},"message":"[SPARK-8841] [SQL] Fix partition pruning percentage log message\n\nWhen pruning partitions for a query plan, a message is logged indicating what how many partitions were selected based on predicate criteria, and what percent were pruned.\n\nThe current release erroneously uses `1 - total/selected` to compute this quantity, leading to nonsense messages like \"pruned -1000% partitions\". The fix is simple and obvious.\n\nAuthor: Steve Lindemann <steve.lindemann@engineersgatelp.com>\n\nCloses #7227 from srlindemann/master and squashes the following commits:\n\nc788061 [Steve Lindemann] fix percentPruned log message","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/39e4e7e4d89077a637c4cad3a986e0e3447d1ae7"},{"sha":"293225e0cd9318ad368dde30ac6a17725d33ebb6","author":{"email":"764d94639167f1ef823b4f9b0508bb82cdb078f2@gmail.com","name":"Daniel Emaasit (PhD Student)"},"message":"[SPARK-8124] [SPARKR] Created more examples on SparkR DataFrames\n\nHere are more examples on SparkR DataFrames including creating a Spark Contect and a SQL\ncontext, loading data and simple data manipulation.\n\nAuthor: Daniel Emaasit (PhD Student) <daniel.emaasit@gmail.com>\n\nCloses #6668 from Emaasit/dan-dev and squashes the following commits:\n\n3a97867 [Daniel Emaasit (PhD Student)] Used fewer rows for createDataFrame\nf7227f9 [Daniel Emaasit (PhD Student)] Using command line arguments\na550f70 [Daniel Emaasit (PhD Student)] Used base R functions\n33f9882 [Daniel Emaasit (PhD Student)] Renamed file\nb6603e3 [Daniel Emaasit (PhD Student)] changed \"Describe\" function to \"describe\"\n90565dd [Daniel Emaasit (PhD Student)] Deleted the getting-started file\nb95a103 [Daniel Emaasit (PhD Student)] Deleted this file\ncc55cd8 [Daniel Emaasit (PhD Student)] combined all the code into one .R file\nc6933af [Daniel Emaasit (PhD Student)] changed variable name to SQLContext\n8e0fe14 [Daniel Emaasit (PhD Student)] provided two options for creating DataFrames\n2653573 [Daniel Emaasit (PhD Student)] Updates to a comment and variable name\n275b787 [Daniel Emaasit (PhD Student)] Added the Apache License at the top of the file\n2e8f724 [Daniel Emaasit (PhD Student)] Added the Apache License at the top of the file\n486f44e [Daniel Emaasit (PhD Student)] Added the Apache License at the file\nd705112 [Daniel Emaasit (PhD Student)] Created more examples on SparkR DataFrames","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/293225e0cd9318ad368dde30ac6a17725d33ebb6"},{"sha":"0e194645f42be0d6ac9b5a712f8fc1798418736d","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-8837][SPARK-7114][SQL] support using keyword in column name\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #7237 from cloud-fan/parser and squashes the following commits:\n\ne7b49bb [Wenchen Fan] support using keyword in column name","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/0e194645f42be0d6ac9b5a712f8fc1798418736d"},{"sha":"57c72fcce75907c08a1ae53a0d85447176fc3c69","author":{"email":"4abe651b51ddbbd99d2cc1417836a36f8d51cdea@gmail.com","name":"Dirceu Semighini Filho"},"message":"Small update in the readme file\n\nJust change the attribute from -PsparkR to -Psparkr\n\nAuthor: Dirceu Semighini Filho <dirceu.semighini@gmail.com>\n\nCloses #7242 from dirceusemighini/patch-1 and squashes the following commits:\n\nfad5991 [Dirceu Semighini Filho] Small update in the readme file","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/57c72fcce75907c08a1ae53a0d85447176fc3c69"},{"sha":"37e4d92142a6309e2df7d36883e0c7892c3d792d","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-8784] [SQL] Add Python API for hex and unhex\n\nAdd Python API for hex/unhex,  also cleanup Hex/Unhex\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #7223 from davies/hex and squashes the following commits:\n\n6f1249d [Davies Liu] no explicit rule to cast string into binary\n711a6ed [Davies Liu] fix test\nf9fe5a3 [Davies Liu] Merge branch 'master' of github.com:apache/spark into hex\nf032fbb [Davies Liu] Merge branch 'hex' of github.com:davies/spark into hex\n49e325f [Davies Liu] Merge branch 'master' of github.com:apache/spark into hex\nb31fc9a [Davies Liu] Update math.scala\n25156b7 [Davies Liu] address comments and fix test\nc3af78c [Davies Liu] address commments\n1a24082 [Davies Liu] Add Python API for hex and unhex","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/37e4d92142a6309e2df7d36883e0c7892c3d792d"},{"sha":"2471c0bf7f463bb144b44a2e51c0f363e71e099d","author":{"email":"156a0858123d9f3ef7d92e126f24abc580e2113c@eecs.berkeley.edu","name":"kai"},"message":"[SPARK-4485] [SQL] 1) Add broadcast hash outer join, (2) Fix SparkPlanTest\n\nThis pull request\n(1) extracts common functions used by hash outer joins and put it in interface HashOuterJoin\n(2) adds ShuffledHashOuterJoin and BroadcastHashOuterJoin\n(3) adds test cases for shuffled and broadcast hash outer join\n(3) makes SparkPlanTest to support binary or more complex operators, and fixes bugs in plan composition in SparkPlanTest\n\nAuthor: kai <kaizeng@eecs.berkeley.edu>\n\nCloses #7162 from kai-zeng/outer and squashes the following commits:\n\n3742359 [kai] Fix not-serializable exception for code-generated keys in broadcasted relations\n14e4bf8 [kai] Use CanBroadcast in broadcast outer join planning\ndc5127e [kai] code style fixes\nb5a4efa [kai] (1) Add broadcast hash outer join, (2) Fix SparkPlanTest","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/2471c0bf7f463bb144b44a2e51c0f363e71e099d"},{"sha":"132e7fca129be8f00ba429a51bcef60abb2eed6d","author":{"email":"c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","name":"Daoyuan Wang"},"message":"[MINOR] [SQL] remove unused code in Exchange\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #7234 from adrian-wang/exchangeclean and squashes the following commits:\n\nb093ec9 [Daoyuan Wang] remove unused code","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/132e7fca129be8f00ba429a51bcef60abb2eed6d"},{"sha":"9ff203346ca4decf2999e33bfb8c400ec75313e6","author":{"email":"e7f91337be682761f64d21f25f9cc67a2ed69aba@appier.com","name":"Wisely Chen"},"message":"[SPARK-8656] [WEBUI] Fix the webUI and JSON API number is not synced\n\nSpark standalone master web UI show \"Alive Workers\" total core, total used cores and \"Alive workers\" total memory, memory used.\nBut the JSON API page \"http://MASTERURL:8088/json\" shows \"ALL workers\"  core, memory number.\nThis webUI data is not sync with the JSON API.\nThe proper way is to sync the number with webUI and JSON API.\n\nAuthor: Wisely Chen <wiselychen@appier.com>\n\nCloses #7038 from thegiive/SPARK-8656 and squashes the following commits:\n\n9e54bf0 [Wisely Chen] Change variable name to camel case\n2c8ea89 [Wisely Chen] Change some styling and add local variable\n431d2b0 [Wisely Chen] Worker List should contain DEAD node also\n8b3b8e8 [Wisely Chen] [SPARK-8656] Fix the webUI and JSON API number is not synced","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/9ff203346ca4decf2999e33bfb8c400ec75313e6"},{"sha":"1165b17d24cdf1dbebb2faca14308dfe5c2a652c","author":{"email":"0b10d4c8e8228fad2615bd1d0ee687d94f12cf9b@brightcove.com","name":"Ankur Chauhan"},"message":"[SPARK-6707] [CORE] [MESOS] Mesos Scheduler should allow the user to specify constraints based on slave attributes\n\nCurrently, the mesos scheduler only looks at the 'cpu' and 'mem' resources when trying to determine the usablility of a resource offer from a mesos slave node. It may be preferable for the user to be able to ensure that the spark jobs are only started on a certain set of nodes (based on attributes).\n\nFor example, If the user sets a property, let's say `spark.mesos.constraints` is set to `tachyon=true;us-east-1=false`, then the resource offers will be checked to see if they meet both these constraints and only then will be accepted to start new executors.\n\nAuthor: Ankur Chauhan <achauhan@brightcove.com>\n\nCloses #5563 from ankurcha/mesos_attribs and squashes the following commits:\n\n902535b [Ankur Chauhan] Fix line length\nd83801c [Ankur Chauhan] Update code as per code review comments\n8b73f2d [Ankur Chauhan] Fix imports\nc3523e7 [Ankur Chauhan] Added docs\n1a24d0b [Ankur Chauhan] Expand scope of attributes matching to include all data types\n482fd71 [Ankur Chauhan] Update access modifier to private[this] for offer constraints\n5ccc32d [Ankur Chauhan] Fix nit pick whitespace\n1bce782 [Ankur Chauhan] Fix nit pick whitespace\nc0cbc75 [Ankur Chauhan] Use offer id value for debug message\n7fee0ea [Ankur Chauhan] Add debug statements\nfc7eb5b [Ankur Chauhan] Fix import codestyle\n00be252 [Ankur Chauhan] Style changes as per code review comments\n662535f [Ankur Chauhan] Incorporate code review comments + use SparkFunSuite\nfdc0937 [Ankur Chauhan] Decline offers that did not meet criteria\n67b58a0 [Ankur Chauhan] Add documentation for spark.mesos.constraints\n63f53f4 [Ankur Chauhan] Update codestyle - uniform style for config values\n02031e4 [Ankur Chauhan] Fix scalastyle warnings in tests\nc09ed84 [Ankur Chauhan] Fixed the access modifier on offerConstraints val to private[mesos]\n0c64df6 [Ankur Chauhan] Rename overhead fractions to memory_*, fix spacing\n8cc1e8f [Ankur Chauhan] Make exception message more explicit about the source of the error\naddedba [Ankur Chauhan] Added test case for malformed constraint string\nec9d9a6 [Ankur Chauhan] Add tests for parse constraint string\n72fe88a [Ankur Chauhan] Fix up tests + remove redundant method override, combine utility class into new mesos scheduler util trait\n92b47fd [Ankur Chauhan] Add attributes based constraints support to MesosScheduler","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/1165b17d24cdf1dbebb2faca14308dfe5c2a652c"}]}