{"push_id":567817737,"size":27,"distinct_size":27,"ref":"refs/heads/master","head":"1390e56fa86d07b588f0c3a1a2cc25ab5bb0db67","before":"80f3bcb58f836cfe1829c85bdd349c10525c8a5e","commits":[{"sha":"fb6c0cbac414f3c43dec2ab886ca8a1097b781f7","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[HOTFIX] Fix test build break in ExecutorAllocationManagerSuite.\n\nThis was caused because #3486 added a new field to ExecutorInfo and #4369\nadded new tests that created ExecutorInfos.  These patches were merged in\nquick succession and were never tested together, hence the compilation error.","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/fb6c0cbac414f3c43dec2ab886ca8a1097b781f7"},{"sha":"af2a2a263ac5d890e84d012b75fcb50e02c9ede8","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-4361][Doc] Add more docs for Hadoop Configuration\n\nI'm trying to point out reusing a Configuration in these APIs is dangerous. Any better idea?\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #3225 from zsxwing/SPARK-4361 and squashes the following commits:\n\nfe4e3d5 [zsxwing] Add more docs for Hadoop Configuration","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/af2a2a263ac5d890e84d012b75fcb50e02c9ede8"},{"sha":"d34f79c8db79ae461fadae190446ebc19091bec9","author":{"email":"7ea8d8bc92d239699affc4b9ea1759f960ddbaa7@huawei.com","name":"WangTaoTheTonic"},"message":"[SPARK-2945][YARN][Doc]add doc for spark.executor.instances\n\nhttps://issues.apache.org/jira/browse/SPARK-2945\n\nspark.executor.instances works. As this JIRA recommended, we should add docs for this common config.\n\nAuthor: WangTaoTheTonic <wangtao111@huawei.com>\n\nCloses #4350 from WangTaoTheTonic/SPARK-2945 and squashes the following commits:\n\n4c3913a [WangTaoTheTonic] not compatible with dynamic allocation\n5fa9c46 [WangTaoTheTonic] add doc for spark.executor.instances","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/d34f79c8db79ae461fadae190446ebc19091bec9"},{"sha":"70e5b030a78ddfdcc8c9eee568009f277dee0872","author":{"email":"fbfd45890515f7eeeeabf87d7a80d3449c3bce83@gmail.com","name":"Nicholas Chammas"},"message":"[SPARK-5628] Add version option to spark-ec2\n\nEvery proper command line tool should include a `--version` option or something similar.\n\nThis PR adds this to `spark-ec2` using the standard functionality provided by `optparse`.\n\nOne thing we don't do here is follow the Python convention of setting `__version__`, since it seems awkward given how `spark-ec2` is laid out.\n\nAuthor: Nicholas Chammas <nicholas.chammas@gmail.com>\n\nCloses #4414 from nchammas/spark-ec2-show-version and squashes the following commits:\n\n914cab5 [Nicholas Chammas] add version info","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/70e5b030a78ddfdcc8c9eee568009f277dee0872"},{"sha":"d433816157bb3ae1f0fbe44efec43a0c906d9f82","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-5650][SQL] Support optional 'FROM' clause\n\nIn Hive, 'FROM' clause is optional. This pr supports it.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #4426 from viirya/optional_from and squashes the following commits:\n\nfe81f31 [Liang-Chi Hsieh] Support optional 'FROM' clause.","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/d433816157bb3ae1f0fbe44efec43a0c906d9f82"},{"sha":"500dc2b4b3136029457e708859fe27da93b1f9e8","author":{"email":"059b8b880f8441509ec8a65b50b4c6ae74ebea76@meisch.ch","name":"Tobias Schlatter"},"message":"[SPARK-5640] Synchronize ScalaReflection where necessary\n\nAuthor: Tobias Schlatter <tobias@meisch.ch>\n\nCloses #4431 from gzm0/sync-scala-refl and squashes the following commits:\n\nc5da21e [Tobias Schlatter] [SPARK-5640] Synchronize ScalaReflection where necessary","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/500dc2b4b3136029457e708859fe27da93b1f9e8"},{"sha":"a958d60975147fb1afc76fcbd80f65ac8d78759a","author":{"email":"62cbe0ac3409a518be51a2793bae027cd01385b1@huawei.com","name":"q00251598"},"message":"[SPARK-5619][SQL] Support 'show roles' in HiveContext\n\nAuthor: q00251598 <qiyadong@huawei.com>\n\nCloses #4397 from watermen/SPARK-5619 and squashes the following commits:\n\nf819b6c [q00251598] Support show roles in HiveContext.","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/a958d60975147fb1afc76fcbd80f65ac8d78759a"},{"sha":"0b7eb3f3b700080bf6cb810d092709a8a468e5db","author":{"email":"f4fb2fdc15585ce98eef9c03d7a86d94a45d60d0@126.com","name":"OopsOutOfMemory"},"message":"[SPARK-5324][SQL] Results of describe can't be queried\n\nMake below code works.\n```\nsql(\"DESCRIBE test\").registerTempTable(\"describeTest\")\nsql(\"SELECT * FROM describeTest\").collect()\n```\n\nAuthor: OopsOutOfMemory <victorshengli@126.com>\nAuthor: Sheng, Li <OopsOutOfMemory@users.noreply.github.com>\n\nCloses #4249 from OopsOutOfMemory/desc_query and squashes the following commits:\n\n6fee13d [OopsOutOfMemory] up-to-date\ne71430a [Sheng, Li] Update HiveOperatorQueryableSuite.scala\n3ba1058 [OopsOutOfMemory] change to default argument\naac7226 [OopsOutOfMemory] Merge branch 'master' into desc_query\n68eb6dd [OopsOutOfMemory] Merge branch 'desc_query' of github.com:OopsOutOfMemory/spark into desc_query\n354ad71 [OopsOutOfMemory] query describe command\nd541a35 [OopsOutOfMemory] refine test suite\ne1da481 [OopsOutOfMemory] refine test suite\na780539 [OopsOutOfMemory] Merge branch 'desc_query' of github.com:OopsOutOfMemory/spark into desc_query\n0015f82 [OopsOutOfMemory] code style\ndd0aaef [OopsOutOfMemory] code style\nc7d606d [OopsOutOfMemory] rename test suite\n75f2342 [OopsOutOfMemory] refine code and test suite\nf942c9b [OopsOutOfMemory] initial\n11559ae [OopsOutOfMemory] code style\nc5fdecf [OopsOutOfMemory] code style\naeaea5f [OopsOutOfMemory] rename test suite\nac2c3bb [OopsOutOfMemory] refine code and test suite\n544573e [OopsOutOfMemory] initial","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/0b7eb3f3b700080bf6cb810d092709a8a468e5db"},{"sha":"3eccf29ce061559c86e6f7338851932fc89a9afd","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-5595][SPARK-5603][SQL] Add a rule to do PreInsert type casting and field renaming and invalidating in memory cache after INSERT\n\nThis PR adds a rule to Analyzer that will add preinsert data type casting and field renaming to the select clause in an `INSERT INTO/OVERWRITE` statement. Also, with the change of this PR, we always invalidate our in memory data cache after inserting into a BaseRelation.\n\ncc marmbrus liancheng\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #4373 from yhuai/insertFollowUp and squashes the following commits:\n\n08237a7 [Yin Huai] Merge remote-tracking branch 'upstream/master' into insertFollowUp\n316542e [Yin Huai] Doc update.\nc9ccfeb [Yin Huai] Revert a unnecessary change.\n84aecc4 [Yin Huai] Address comments.\n1951fe1 [Yin Huai] Merge remote-tracking branch 'upstream/master'\nc18da34 [Yin Huai] Invalidate cache after insert.\n727f21a [Yin Huai] Preinsert casting and renaming.","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/3eccf29ce061559c86e6f7338851932fc89a9afd"},{"sha":"b62c35245a6c9bee6a335aa5dcfcca2547e85406","author":{"email":"f4fb2fdc15585ce98eef9c03d7a86d94a45d60d0@126.com","name":"OopsOutOfMemory"},"message":"[SQL][HiveConsole][DOC] HiveConsole `correct hiveconsole imports`\n\nSorry for that PR #4330 has some mistakes.\n\nI correct it....  so it works correctly now.\n\nAuthor: OopsOutOfMemory <victorshengli@126.com>\n\nCloses #4389 from OopsOutOfMemory/doc and squashes the following commits:\n\n843eed9 [OopsOutOfMemory] correct hiveconsole imports","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/b62c35245a6c9bee6a335aa5dcfcca2547e85406"},{"sha":"bc36356080e3b52aaf61fc1e6b204146ab96d29f","author":{"email":"f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","name":"wangfei"},"message":"[SQL][Minor] Remove cache keyword in SqlParser\n\nSince cache keyword already defined in `SparkSQLParser` and `SqlParser` of catalyst is a more general parser which should not cover keywords related to underlying compute engine, to remove  cache keyword in  `SqlParser`.\n\nAuthor: wangfei <wangfei1@huawei.com>\n\nCloses #4393 from scwf/remove-cache-keyword and squashes the following commits:\n\n10ade16 [wangfei] remove cache keyword in sql parser","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/bc36356080e3b52aaf61fc1e6b204146ab96d29f"},{"sha":"4793c8402a19afe4df51129a7f99e07494a76af2","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-5278][SQL] Introduce UnresolvedGetField and complete the check of ambiguous reference to fields\n\nWhen the `GetField` chain(`a.b.c.d.....`) is interrupted by `GetItem` like `a.b[0].c.d....`, then the check of ambiguous reference to fields is broken.\nThe reason is that: for something like `a.b[0].c.d`, we first parse it to `GetField(GetField(GetItem(Unresolved(\"a.b\"), 0), \"c\"), \"d\")`. Then in `LogicalPlan#resolve`, we resolve `\"a.b\"` and build a `GetField` chain from bottom(the relation). But for the 2 outer `GetFiled`, we have to resolve them in `Analyzer` or do it in `GetField` lazily, check data type of child, search needed field, etc. which is similar to what we have done in `LogicalPlan#resolve`.\nSo in this PR, the fix is just copy the same logic in `LogicalPlan#resolve` to `Analyzer`, which is simple and quick, but I do suggest introduce `UnresolvedGetFiled` like I explained in https://github.com/apache/spark/pull/2405.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #4068 from cloud-fan/simple and squashes the following commits:\n\na6857b5 [Wenchen Fan] fix import order\n8411c40 [Wenchen Fan] use UnresolvedGetField","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/4793c8402a19afe4df51129a7f99e07494a76af2"},{"sha":"3d3ecd77411bfdd1c72be51616c46a6caf839be2","author":{"email":"f4fb2fdc15585ce98eef9c03d7a86d94a45d60d0@126.com","name":"OopsOutOfMemory"},"message":"[SPARK-5586][Spark Shell][SQL] Make `sqlContext` available in spark shell\n\nResult is like this\n```\n15/02/05 13:41:22 INFO SparkILoop: Created spark context..\nSpark context available as sc.\n15/02/05 13:41:22 INFO SparkILoop: Created sql context..\nSQLContext available as sqlContext.\n\nscala> sq\nsql          sqlContext   sqlParser    sqrt\n```\n\nAuthor: OopsOutOfMemory <victorshengli@126.com>\n\nCloses #4387 from OopsOutOfMemory/sqlContextInShell and squashes the following commits:\n\nc7f5203 [OopsOutOfMemory] auto-import sql() function\ne160697 [OopsOutOfMemory] Merge branch 'sqlContextInShell' of https://github.com/OopsOutOfMemory/spark into sqlContextInShell\n37c0a16 [OopsOutOfMemory] auto detect hive support\na9c59d9 [OopsOutOfMemory] rename and reduce range of imports\n6b9e309 [OopsOutOfMemory] Merge branch 'master' into sqlContextInShell\ncae652f [OopsOutOfMemory] make sqlContext available in spark shell","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/3d3ecd77411bfdd1c72be51616c46a6caf839be2"},{"sha":"0f3a36071a44e986d97981032d5b192477b38bbd","author":{"email":"2f805403620fd0413e09d4227afe73325dc01d5f@gmail.com","name":"GenTang"},"message":"[SPARK-4983] Insert waiting time before tagging EC2 instances\n\nThe boto API doesn't support tag EC2 instances in the same call that launches them.\nWe add a five-second wait so EC2 has enough time to propagate the information so that\nthe tagging can succeed.\n\nAuthor: GenTang <gen.tang86@gmail.com>\nAuthor: Gen TANG <gen.tang86@gmail.com>\n\nCloses #3986 from GenTang/spark-4983 and squashes the following commits:\n\n13e257d [Gen TANG] modification of comments\n47f06755 [GenTang] print the information\nab7a931 [GenTang] solve the issus spark-4983 by inserting waiting time\n3179737 [GenTang] Revert \"handling exceptions about adding tags to ec2\"\n6a8b53b [GenTang] Revert \"the improvement of exception handling\"\n13e97a6 [GenTang] Revert \"typo\"\n63fd360 [GenTang] typo\n692fc2b [GenTang] the improvement of exception handling\n6adcf6d [GenTang] handling exceptions about adding tags to ec2","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/0f3a36071a44e986d97981032d5b192477b38bbd"},{"sha":"65181b7512c2192311705dfdfad588a80d3a6fde","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[HOTFIX][MLLIB] fix a compilation error with java 6\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #4442 from mengxr/java6-fix and squashes the following commits:\n\n2098500 [Xiangrui Meng] fix a compilation error with java 6","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/65181b7512c2192311705dfdfad588a80d3a6fde"},{"sha":"b3872e00d155939e40366debda635fc3fb12cc73","author":{"email":"d738347005911ea562051ad08f45b5365232ab6e@magnetic.com","name":"Vladimir Vladimirov"},"message":"SPARK-5633 pyspark saveAsTextFile support for compression codec\n\nSee https://issues.apache.org/jira/browse/SPARK-5633 for details\n\nAuthor: Vladimir Vladimirov <vladimir.vladimirov@magnetic.com>\n\nCloses #4403 from smartkiwi/master and squashes the following commits:\n\n94c014e [Vladimir Vladimirov] SPARK-5633 pyspark saveAsTextFile support for compression codec","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/b3872e00d155939e40366debda635fc3fb12cc73"},{"sha":"ca66159a4f30d65fa4cd32dbf3ff23978cb7f99b","author":{"email":"4250ae0beae190a9e45365b4e2d96f9f581eef84@guavus.com","name":"Kashish Jain"},"message":"SPARK-5613: Catch the ApplicationNotFoundException exception to avoid thread from getting killed on yarn restart.\n\n[SPARK-5613] Added a  catch block to catch the ApplicationNotFoundException. Without this catch block the thread gets killed on occurrence of this exception. This Exception occurs when yarn restarts and tries to find an application id for a spark job which got interrupted due to yarn getting stopped.\nSee the stacktrace in the bug for more details.\n\nAuthor: Kashish Jain <kashish.jain@guavus.com>\n\nCloses #4392 from kasjain/branch-1.2 and squashes the following commits:\n\n4831000 [Kashish Jain] SPARK-5613: Catch the ApplicationNotFoundException exception to avoid thread from getting killed on yarn restart.","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/ca66159a4f30d65fa4cd32dbf3ff23978cb7f99b"},{"sha":"5687bab8fdfdc5345b8c5b9be8d4595299005fc8","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-5600] [core] Clean up FsHistoryProvider test, fix app sort order.\n\nClean up some test setup code to remove duplicate instantiation of the\nprovider. Also make sure unfinished apps are sorted correctly.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #4370 from vanzin/SPARK-5600 and squashes the following commits:\n\n0d048d5 [Marcelo Vanzin] Cleanup test code a bit.\n2585119 [Marcelo Vanzin] Review feedback.\n8b97544 [Marcelo Vanzin] Merge branch 'master' into SPARK-5600\nbe979e9 [Marcelo Vanzin] Merge branch 'master' into SPARK-5600\n298371c [Marcelo Vanzin] [SPARK-5600] [core] Clean up FsHistoryProvider test, fix app sort order.","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/5687bab8fdfdc5345b8c5b9be8d4595299005fc8"},{"sha":"57961567ef104efb3174e67d762c5d9d6263b800","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[HOTFIX] Fix the maven build after adding sqlContext to spark-shell\n\nFollow up to #4387 to fix the build break.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #4443 from marmbrus/fixMaven and squashes the following commits:\n\n1eeba7d [Michael Armbrust] try again\n7f5fb15 [Michael Armbrust] [HOTFIX] Fix the maven build after adding sqlContext to spark-shell","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/57961567ef104efb3174e67d762c5d9d6263b800"},{"sha":"dcd1e42d6b6ac08d2c0736bf61a15f515a1f222b","author":{"email":"e1a8399e36effe75a124102eb2a2bdfc82e0aab4@cloudera.com","name":"Kostas Sakellis"},"message":"[SPARK-4874] [CORE] Collect record count metrics\n\nCollects record counts for both Input/Output and Shuffle Metrics. For the input/output metrics, it just appends the counter every time the iterators get accessed.\n\nFor shuffle on the write side, we count the metrics post aggregation (after a map side combine) and on the read side we count the metrics pre aggregation. This allows both the bytes read/written metrics and the records read/written to line up.\n\nFor backwards compatibility, if we deserialize an older event that doesn't have record metrics, we set the metric to -1.\n\nAuthor: Kostas Sakellis <kostas@cloudera.com>\n\nCloses #4067 from ksakellis/kostas-spark-4874 and squashes the following commits:\n\nbd919be [Kostas Sakellis] Changed 'Records Read' in shuffleReadMetrics json output to 'Total Records Read'\ndad4d57 [Kostas Sakellis] Add a comment and check to BlockObjectWriter so that it cannot be reopend.\n6f236a1 [Kostas Sakellis] Renamed _recordsWritten in ShuffleWriteMetrics to be more consistent\n70620a0 [Kostas Sakellis] CR Feedback\n17faa3a [Kostas Sakellis] Removed AtomicLong in favour of using Long\nb6f9923 [Kostas Sakellis] Merge AfterNextInterceptingIterator with InterruptableIterator to save a function call\n46c8186 [Kostas Sakellis] Combined Bytes and # records into one column\n57551c1 [Kostas Sakellis] Conforms to SPARK-3288\n6cdb44e [Kostas Sakellis] Removed the generic InterceptingIterator and repalced it with specific implementation\n1aa273c [Kostas Sakellis] CR Feedback\n1bb78b1 [Kostas Sakellis] [SPARK-4874] [CORE] Collect record count metrics","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/dcd1e42d6b6ac08d2c0736bf61a15f515a1f222b"}]}