{"push_id":631272319,"size":49,"distinct_size":41,"ref":"refs/heads/string","head":"6ce7c0ba608526e852058ce4edf68db7e831339c","before":"b04a19c8978abaac2a68173e3f412728d7e826e1","commits":[{"sha":"b5c51c8df480f1a82a82e4d597d8eea631bffb4e","author":{"email":"399a71a219a2bbf15a9ad52c797d50eb921e7e28@gmail.com","name":"Davies Liu"},"message":"[SPARK-3074] [PySpark] support groupByKey() with single huge key\n\nThis patch change groupByKey() to use external sort based approach, so it can support single huge key.\n\nFor example, it can group by a dataset including one hot key with 40 millions values (strings), using 500M memory for Python worker, finished in about 2 minutes. (it will need 6G memory in hash based approach).\n\nDuring groupByKey(), it will do in-memory groupBy first. If the dataset can not fit in memory, then data will be partitioned by hash. If one partition still can not fit in memory, it will switch to sort based groupBy().\n\nAuthor: Davies Liu <davies.liu@gmail.com>\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #1977 from davies/groupby and squashes the following commits:\n\naf3713a [Davies Liu] make sure it's iterator\n67772dd [Davies Liu] fix tests\ne78c15c [Davies Liu] address comments\n0b0fde8 [Davies Liu] address comments\n0dcf320 [Davies Liu] address comments, rollback changes in ResultIterable\ne3b8eab [Davies Liu] fix narrow dependency\n2a1857a [Davies Liu] typo\nd2f053b [Davies Liu] add repr for FlattedValuesSerializer\nc6a2f8d [Davies Liu] address comments\n9e2df24 [Davies Liu] Merge branch 'master' of github.com:apache/spark into groupby\n2b9c261 [Davies Liu] fix typo in comments\n70aadcd [Davies Liu] Merge branch 'master' of github.com:apache/spark into groupby\na14b4bd [Davies Liu] Merge branch 'master' of github.com:apache/spark into groupby\nab5515b [Davies Liu] Merge branch 'master' into groupby\n651f891 [Davies Liu] simplify GroupByKey\n1578f2e [Davies Liu] Merge branch 'master' of github.com:apache/spark into groupby\n1f69f93 [Davies Liu] fix tests\n0d3395f [Davies Liu] Merge branch 'master' of github.com:apache/spark into groupby\n341f1e0 [Davies Liu] add comments, refactor\n47918b8 [Davies Liu] remove unused code\n6540948 [Davies Liu] address comments:\n17f4ec6 [Davies Liu] Merge branch 'master' of github.com:apache/spark into groupby\n4d4bc86 [Davies Liu] bugfix\n8ef965e [Davies Liu] Merge branch 'master' into groupby\nfbc504a [Davies Liu] Merge branch 'master' into groupby\n779ed03 [Davies Liu] fix merge conflict\n2c1d05b [Davies Liu] refactor, minor turning\nb48cda5 [Davies Liu] Merge branch 'master' into groupby\n85138e6 [Davies Liu] Merge branch 'master' into groupby\nacd8e1b [Davies Liu] fix memory when groupByKey().count()\n905b233 [Davies Liu] Merge branch 'sort' into groupby\n1f075ed [Davies Liu] Merge branch 'master' into sort\n4b07d39 [Davies Liu] compress the data while spilling\n0a081c6 [Davies Liu] Merge branch 'master' into groupby\nf157fe7 [Davies Liu] Merge branch 'sort' into groupby\neb53ca6 [Davies Liu] Merge branch 'master' into sort\nb2dc3bf [Davies Liu] Merge branch 'sort' into groupby\n644abaf [Davies Liu] add license in LICENSE\n19f7873 [Davies Liu] improve tests\n11ba318 [Davies Liu] typo\n085aef8 [Davies Liu] Merge branch 'master' into groupby\n3ee58e5 [Davies Liu] switch to sort based groupBy, based on size of data\n1ea0669 [Davies Liu] choose sort based groupByKey() automatically\nb40bae7 [Davies Liu] bugfix\nefa23df [Davies Liu] refactor, add spark.shuffle.sort=False\n250be4e [Davies Liu] flatten the combined values when dumping into disks\nd05060d [Davies Liu] group the same key before shuffle, reduce the comparison during sorting\n083d842 [Davies Liu] sorted based groupByKey()\n55602ee [Davies Liu] use external sort in sortBy() and sortByKey()","distinct":false,"url":"https://api.github.com/repos/davies/spark/commits/b5c51c8df480f1a82a82e4d597d8eea631bffb4e"},{"sha":"e2360810f50de77f79d372cc9b46db117d451cfc","author":{"email":"8757c4a255384de9fc2175f17a84f948e7081d30@gmail.com","name":"MechCoder"},"message":"[SPARK-6577] [MLlib] [PySpark] SparseMatrix should be supported in PySpark\n\nSupporting of SparseMatrix in PySpark.\n\nAuthor: MechCoder <manojkumarsivaraj334@gmail.com>\n\nCloses #5355 from MechCoder/spark-6577 and squashes the following commits:\n\n7492190 [MechCoder] More readable code for densifying\nea2c54b [MechCoder] Check bounds for indexing\n454ef2c [MechCoder] Made the following changes 1. Used convert_to_array for array conversion. 2. Used F order for toArray 3. Minor improvements in speed.\ndb76caf [MechCoder] Add support for CSR matrix\n29653e7 [MechCoder] Renamed indices to rowIndices and indptr to colPtrs\nb6384fe [MechCoder] [SPARK-6577] SparseMatrix should be supported in PySpark","distinct":false,"url":"https://api.github.com/repos/davies/spark/commits/e2360810f50de77f79d372cc9b46db117d451cfc"},{"sha":"3290d2d13bb4bd875aec14425c8e3766f9cc644b","author":{"email":"79478dbbc8641d1d6545f348f655464aefbda99b@intel.com","name":"jerryshao"},"message":"[SPARK-6211][Streaming] Add Python Kafka API unit test\n\nRefactor the Kafka unit test and add Python API support. CC tdas davies please help to review, thanks a lot.\n\nAuthor: jerryshao <saisai.shao@intel.com>\nAuthor: Saisai Shao <saisai.shao@intel.com>\n\nCloses #4961 from jerryshao/SPARK-6211 and squashes the following commits:\n\nee4b919 [jerryshao] Fixed newly merged issue\n82c756e [jerryshao] Address the comments\n92912d1 [jerryshao] Address the commits\n0708bb1 [jerryshao] Fix rebase issue\n40b47a3 [Saisai Shao] Style fix\nf889657 [Saisai Shao] Update the code according\n8a2f3e2 [jerryshao] Address the issues\n0f1b7ce [jerryshao] Still fix the bug\n61a04f0 [jerryshao] Fix bugs and address the issues\n64d9877 [jerryshao] Fix rebase bugs\n8ad442f [jerryshao] Add kafka-assembly in run-tests\n6020b00 [jerryshao] Add more debug info in Shell\n8102d6e [jerryshao] Fix bug in Jenkins test\nfde1213 [jerryshao] Code style changes\n5536f95 [jerryshao] Refactor the Kafka unit test and add Python Kafka unittest support","distinct":false,"url":"https://api.github.com/repos/davies/spark/commits/3290d2d13bb4bd875aec14425c8e3766f9cc644b"},{"sha":"18ca089bed41ce3e87deeb14206317863518c12c","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-6766][Streaming] Fix issue about StreamingListenerBatchSubmitted and StreamingListenerBatchStarted\n\nThis PR includes:\n\n1. Send `StreamingListenerBatchSubmitted` when `JobSet` is submitted\n1. Fix `StreamingListenerBatchStarted.batchInfo.processingStartTime`\n1. Fix a type: `completedaBatchInfos` -> `completedBatchInfos`\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #5414 from zsxwing/SPARK-6766 and squashes the following commits:\n\n2f85060 [zsxwing] Update tests\nca0955b [zsxwing] Combine unit tests\n79b4fed [zsxwing] Add StreamingJobProgressListenerSuite to test StreamingJobProgressListener\nfc3a2a1 [zsxwing] Add unit tests for SPARK-6766\n74aed99 [zsxwing] Refactor as per TD's suggestion\n493f978 [zsxwing] Send StreamingListenerBatchSubmitted when JobSet is submitted; fix StreamingListenerBatchStarted.batchInfo.processingStartTime; fix a typo","distinct":false,"url":"https://api.github.com/repos/davies/spark/commits/18ca089bed41ce3e87deeb14206317863518c12c"},{"sha":"9f5ed99d644949443d19c4895de6e0ece4be24d0","author":{"email":"7500a5f2e9dbcebe7fe7716edfd0a44e2fd44837@huawei.com","name":"June.He"},"message":"[SPARK-6773][Tests]Fix RAT checks still passed issue when download rat jar failed\n\ncheck -license will passed in next time when rat jar download failed.\n\nAdd 2 step to fix this:\n1. Clean the rat.jar if download failed.\n2. Add a check logic after run rat checking.\n\nAuthor: June.He <jun.hejun@huawei.com>\n\nCloses #5421 from sisihj/rat-issue and squashes the following commits:\n\n4958302 [June.He] delete redundant check and add quoted\n66f7088 [June.He] Fix RAT checks still passed issue when download rat jar failed\n7311e83 [June.He] Fix RAT checks still passed issue when download rat jar failed","distinct":false,"url":"https://api.github.com/repos/davies/spark/commits/9f5ed99d644949443d19c4895de6e0ece4be24d0"},{"sha":"b9baa4cd9f6e9fc58161f79744b6b7729894d920","author":{"email":"415008509078806504352d2554eb2c4dae69e44f@gmail.com","name":"Volodymyr Lyubinets"},"message":"[SQL] [SPARK-6794] Use kryo-based SparkSqlSerializer for GeneralHashedRelation\n\nBenchmarking results: http://pastie.org/private/1dneo1mta5zpsw6gmsoeq\n\nAuthor: Volodymyr Lyubinets <vlyubin@gmail.com>\n\nCloses #5433 from vlyubin/joins and squashes the following commits:\n\nd70c829 [Volodymyr Lyubinets] Addressed review feedback\n527eac6 [Volodymyr Lyubinets] Use kryo-based SparkSqlSerializer for GeneralHashedRelation","distinct":false,"url":"https://api.github.com/repos/davies/spark/commits/b9baa4cd9f6e9fc58161f79744b6b7729894d920"},{"sha":"0375134f42197f2e29aa865a513cda381f0a1445","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"[SPARK-5969][PySpark] Fix descending pyspark.rdd.sortByKey.\n\nThe samples should always be sorted in ascending order, because bisect.bisect_left is used on it. The reverse order of the result is already achieved in rangePartitioner by reversing the found index.\n\nThe current implementation also work, but always uses only two partitions -- the first one and the last one (because the bisect_left return returns either \"beginning\" or \"end\" for a descending sequence).\n\nAuthor: Milan Straka <fox@ucw.cz>\n\nThis patch had conflicts when merged, resolved by\nCommitter: Josh Rosen <joshrosen@databricks.com>\n\nCloses #4761 from foxik/fix-descending-sort and squashes the following commits:\n\n95896b5 [Milan Straka] Add regression test for SPARK-5969.\n5757490 [Milan Straka] Fix descending pyspark.rdd.sortByKey.","distinct":false,"url":"https://api.github.com/repos/davies/spark/commits/0375134f42197f2e29aa865a513cda381f0a1445"},{"sha":"4740d6a158cb4d35408a95265c5b950b9e9628a3","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-6216] [PySpark] check the python version in worker\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #5404 from davies/check_version and squashes the following commits:\n\ne559248 [Davies Liu] add tests\nec33b5f [Davies Liu] check the python version in worker","distinct":false,"url":"https://api.github.com/repos/davies/spark/commits/4740d6a158cb4d35408a95265c5b950b9e9628a3"},{"sha":"68ecdb7f99ae30f7c04c33a47ab7f59a3836f2a4","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-6850] [SparkR] use one partition when we need to compare the whole result\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #5460 from davies/r_test and squashes the following commits:\n\n0a593ce [Davies Liu] use one partition when we need to compare the whole result","distinct":true,"url":"https://api.github.com/repos/davies/spark/commits/68ecdb7f99ae30f7c04c33a47ab7f59a3836f2a4"},{"sha":"23d5f8864f7d665a74b1d38118700139854dbb1c","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-6851][SQL] Create new instance for each converted parquet relation\n\nOtherwise we end up rewriting predicates to be trivially equal (i.e. `a#1 = a#2` -> `a#3 = a#3`), at which point the query is no longer valid.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #5458 from marmbrus/selfJoinParquet and squashes the following commits:\n\n22df77c [Michael Armbrust] [SPARK-6851][SQL] Create new instance for each converted parquet relation","distinct":true,"url":"https://api.github.com/repos/davies/spark/commits/23d5f8864f7d665a74b1d38118700139854dbb1c"},{"sha":"67d06880e47e0324409cf7e5b21db1dcb0107b82","author":{"email":"415008509078806504352d2554eb2c4dae69e44f@gmail.com","name":"Volodymyr Lyubinets"},"message":"[SQL] [SPARK-6620] Speed up toDF() and rdd() functions by constructing converters in ScalaReflection\n\ncc marmbrus\n\nAuthor: Volodymyr Lyubinets <vlyubin@gmail.com>\n\nCloses #5279 from vlyubin/speedup and squashes the following commits:\n\ne75a387 [Volodymyr Lyubinets] Changes to ScalaUDF\n11a20ec [Volodymyr Lyubinets] Avoid creating a tuple\nc327bc9 [Volodymyr Lyubinets] Moved the only remaining function from DataTypeConversions to DateUtils\ndec6802 [Volodymyr Lyubinets] Addresed review feedback\n74301fa [Volodymyr Lyubinets] Addressed review comments\nafa3aa5 [Volodymyr Lyubinets] Minor refactoring, added license, removed debug output\n881dc60 [Volodymyr Lyubinets] Moved to a separate module; addressed review comments; one extra place of usage; changed behaviour for Java\n8cad6e2 [Volodymyr Lyubinets] Addressed review commments\n41b2aa9 [Volodymyr Lyubinets] Creating converters for ScalaReflection stuff, and more","distinct":true,"url":"https://api.github.com/repos/davies/spark/commits/67d06880e47e0324409cf7e5b21db1dcb0107b82"},{"sha":"95a07591b3e23782a7021ef2bbf07a67a1e4a83a","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[Minor][Core] Fix typo\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #5466 from viirya/fix_ShuffleMapTask_typo and squashes the following commits:\n\n2789fd5 [Liang-Chi Hsieh] fix typo.","distinct":true,"url":"https://api.github.com/repos/davies/spark/commits/95a07591b3e23782a7021ef2bbf07a67a1e4a83a"},{"sha":"694aef0d71d2683eaf63cbd1d8e95c2da423b72e","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[hotfix] [build] Make sure JAVA_HOME is set for tests.\n\nThis is needed at least for YARN integration tests, since `$JAVA_HOME` is used to launch the executors.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #5441 from vanzin/yarn-test-test and squashes the following commits:\n\n3eeec30 [Marcelo Vanzin] Use JAVA_HOME when available, java.home otherwise.\nd71f1bb [Marcelo Vanzin] And sbt too.\n6bda399 [Marcelo Vanzin] WIP: Testing to see whether this fixes the yarn test issue.","distinct":true,"url":"https://api.github.com/repos/davies/spark/commits/694aef0d71d2683eaf63cbd1d8e95c2da423b72e"},{"sha":"3ceb810aa8e69bc4abb69cbe713a624cb351cb35","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-6835] [SQL] Fix bug of Hive UDTF in Lateral View (ClassNotFound)\n\n```SQL\nselect key, v from src lateral view stack(3, 1+1, 2+2, 3) d as v;\n```\nWill cause exception\n```\njava.lang.ClassNotFoundException: stack\nat java.net.URLClassLoader$1.run(URLClassLoader.java:366)\nat java.net.URLClassLoader$1.run(URLClassLoader.java:355)\nat java.security.AccessController.doPrivileged(Native Method)\nat java.net.URLClassLoader.findClass(URLClassLoader.java:354)\nat java.lang.ClassLoader.loadClass(ClassLoader.java:425)\nat sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)\nat java.lang.ClassLoader.loadClass(ClassLoader.java:358)\nat org.apache.spark.sql.hive.HiveFunctionWrapper.createFunction(Shim13.scala:148)\nat org.apache.spark.sql.hive.HiveGenericUdtf.function$lzycompute(hiveUdfs.scala:274)\nat org.apache.spark.sql.hive.HiveGenericUdtf.function(hiveUdfs.scala:274)\nat org.apache.spark.sql.hive.HiveGenericUdtf.outputInspector$lzycompute(hiveUdfs.scala:280)\nat org.apache.spark.sql.hive.HiveGenericUdtf.outputInspector(hiveUdfs.scala:280)\nat org.apache.spark.sql.hive.HiveGenericUdtf.outputDataTypes$lzycompute(hiveUdfs.scala:285)\nat org.apache.spark.sql.hive.HiveGenericUdtf.outputDataTypes(hiveUdfs.scala:285)\nat org.apache.spark.sql.hive.HiveGenericUdtf.makeOutput(hiveUdfs.scala:291)\nat org.apache.spark.sql.catalyst.expressions.Generator.output(generators.scala:60)\nat org.apache.spark.sql.catalyst.plans.logical.Generate$$anonfun$2.apply(basicOperators.scala:60)\nat org.apache.spark.sql.catalyst.plans.logical.Generate$$anonfun$2.apply(basicOperators.scala:60)\nat scala.Option.map(Option.scala:145)\nat org.apache.spark.sql.catalyst.plans.logical.Generate.generatorOutput(basicOperators.scala:60)\nat org.apache.spark.sql.catalyst.plans.logical.Generate.output(basicOperators.scala:70)\nat org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveChildren$1.apply(LogicalPlan.scala:117)\nat org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveChildren$1.apply(LogicalPlan.scala:117)\n```\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #5444 from chenghao-intel/hive_udtf and squashes the following commits:\n\n065a98c [Cheng Hao] fix bug of Hive UDTF in Lateral View (ClassNotFound)","distinct":true,"url":"https://api.github.com/repos/davies/spark/commits/3ceb810aa8e69bc4abb69cbe713a624cb351cb35"},{"sha":"198cf2a3fa9babb5e8b7b44da0471b63b9f3ec04","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-6858][SQL] Register Java HashMap for SparkSqlSerializer\n\nSince now kyro serializer is used for `GeneralHashedRelation` whether kyro is enabled or not, it is better to register Java `HashMap` in `SparkSqlSerializer`.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #5465 from viirya/register_hashmap and squashes the following commits:\n\n9062601 [Liang-Chi Hsieh] Register Java HashMap for SparkSqlSerializer.","distinct":true,"url":"https://api.github.com/repos/davies/spark/commits/198cf2a3fa9babb5e8b7b44da0471b63b9f3ec04"},{"sha":"5f7b7cdab41002ad291f0a1ba7777cfff21d0019","author":{"email":"34c796fa99e5c551a03758576354487d76ffb3a4@sap.com","name":"Santiago M. Mola"},"message":"[SPARK-6611][SQL] Add support for INTEGER as synonym of INT.\n\nhttps://issues.apache.org/jira/browse/SPARK-6611\n\nAuthor: Santiago M. Mola <santiago.mola@sap.com>\n\nCloses #5271 from smola/features/integer-parse and squashes the following commits:\n\nf5c1c64 [Santiago M. Mola] [SPARK-6611] Add support for INTEGER as synonym of INT.","distinct":true,"url":"https://api.github.com/repos/davies/spark/commits/5f7b7cdab41002ad291f0a1ba7777cfff21d0019"},{"sha":"6437e7cc3bd405ebd4ad7d8f9c7a5e703652ad36","author":{"email":"34c796fa99e5c551a03758576354487d76ffb3a4@sap.com","name":"Santiago M. Mola"},"message":"[SPARK-6863] Fix formatting on SQL programming guide.\n\nhttps://issues.apache.org/jira/browse/SPARK-6863\n\nAuthor: Santiago M. Mola <santiago.mola@sap.com>\n\nCloses #5472 from smola/fix/sql-docs and squashes the following commits:\n\n42503d4 [Santiago M. Mola] [SPARK-6863] Fix formatting on SQL programming guide.","distinct":true,"url":"https://api.github.com/repos/davies/spark/commits/6437e7cc3bd405ebd4ad7d8f9c7a5e703652ad36"},{"sha":"7dbd37160ff57f80cc7abdcaef95f8c6df20a0f0","author":{"email":"21c6bf6d42817f60587cc5b1dd73f82d8a23dc80@gmail.com","name":"Guancheng (G.C.) Chen"},"message":"[Minor][SQL] Fix typo in sql\n\nIn this PR, \"analyser\" is changed to \"analyzer\" to keep a consistent naming. Some other typos are also fixed.\n\nAuthor: Guancheng (G.C.) Chen <chenguancheng@gmail.com>\n\nCloses #5474 from gchen/sql-typo and squashes the following commits:\n\n70e6e76 [Guancheng (G.C.) Chen] Merge branch 'sql-typo' of github.com:gchen/spark into sql-typo\nfb7a6e2 [Guancheng (G.C.) Chen] fix typo in sql\n37e3da1 [Guancheng (G.C.) Chen] fix type in sql","distinct":true,"url":"https://api.github.com/repos/davies/spark/commits/7dbd37160ff57f80cc7abdcaef95f8c6df20a0f0"},{"sha":"2f53588738e95a2191f9844818e47f0d2ebbfd54","author":{"email":"364864161977e087e6bcddb648b6e71f10d09f2c@huawei.com","name":"haiyang"},"message":"[SPARK-6199] [SQL] Support CTE in HiveContext and SQLContext\n\nAuthor: haiyang <huhaiyang@huawei.com>\n\nCloses #4929 from haiyangsea/cte and squashes the following commits:\n\n220b67d [haiyang] add golden files for cte test\nd3c7681 [haiyang] Merge branch 'master' into cte-repair\n0ba2070 [haiyang] modify code style\n9ce6b58 [haiyang] fix conflict\nff74741 [haiyang] add comment for With plan\n0d56af4 [haiyang] code indention\n776a440 [haiyang] add comments for resolve relation strategy\n2fccd7e [haiyang] add comments for resolve relation strategy\n241bbe2 [haiyang] fix cte problem of view\ne9e1237 [haiyang] fix test case problem\n614182f [haiyang] add test cases for CTE feature\n32e415b [haiyang] add comment\n1cc8c15 [haiyang] support with\n03f1097 [haiyang] support with\ne960099 [haiyang] support with\n9aaa874 [haiyang] support with\n0566978 [haiyang] support with\na99ecd2 [haiyang] support with\nc3fa4c2 [haiyang] support with\n3b6077f [haiyang] support with\n5f8abe3 [haiyang] support with\n4572b05 [haiyang] support with\nf801f54 [haiyang] support with","distinct":true,"url":"https://api.github.com/repos/davies/spark/commits/2f53588738e95a2191f9844818e47f0d2ebbfd54"},{"sha":"1f39a61118184e136f38381a9f3ba0b2d5d589d9","author":{"email":"04bd8ed65586cb5b7e15d141cd94e216e7feda23@gmail.com","name":"lazymam500"},"message":"[Spark-5068][SQL]Fix bug query data when path doesn't exist for HiveContext\n\nThis PR follow up PR #3907 & #3891 & #4356.\nAccording to  marmbrus  liancheng 's comments, I try to use fs.globStatus to retrieve all FileStatus objects under path(s), and then do the filtering locally.\n\n[1]. get pathPattern by path, and put it into pathPatternSet. (hdfs://cluster/user/demo/2016/08/12 -> hdfs://cluster/user/demo/*/*/*)\n[2]. retrieve all FileStatus objects ,and cache them by undating existPathSet.\n[3]. do the filtering locally\n[4]. if we have new pathPattern,do 1,2 step again. (external table maybe have more than one partition pathPattern)\n\nchenghao-intel jeanlyn\n\nAuthor: lazymam500 <lazyman500@gmail.com>\nAuthor: lazyman <lazyman500@gmail.com>\n\nCloses #5059 from lazyman500/SPARK-5068 and squashes the following commits:\n\n5bfcbfd [lazyman] move spark.sql.hive.verifyPartitionPath to SQLConf,fix scala style\ne1d6386 [lazymam500] fix scala style\nf23133f [lazymam500] bug fix\n47e0023 [lazymam500] fix scala style,add config flag,break the chaining\n04c443c [lazyman] SPARK-5068: fix bug when partition path doesn't exists #2\n41f60ce [lazymam500] Merge pull request #1 from apache/master","distinct":true,"url":"https://api.github.com/repos/davies/spark/commits/1f39a61118184e136f38381a9f3ba0b2d5d589d9"}]}