{"push_id":565660309,"size":13,"distinct_size":13,"ref":"refs/heads/master","head":"1fbd124b1bd6159086d8e88b139ce0817af02322","before":"b0c0021953826bccaee818a54afc44e8bdfa8572","commits":[{"sha":"f0500f9fa378d81e4b4038a66a40eee15806b677","author":{"email":"e1f8d9790f67825b23faf6a373ab7c83bc8900f2@apache.org","name":"Hari Shreedharan"},"message":"[SPARK-4707][STREAMING] Reliable Kafka Receiver can lose data if the blo...\n\n...ck generator fails to store data.\n\nThe Reliable Kafka Receiver commits offsets only when events are actually stored, which ensures that on restart we will actually start where we left off. But if the failure happens in the store() call, and the block generator reports an error the receiver does not do anything and will continue reading from the current offset and not the last commit. This means that messages between the last commit and the current offset will be lost.\n\nThis PR retries the store call four times and then stops the receiver with an error message and the last exception that was received from the store.\n\nAuthor: Hari Shreedharan <hshreedharan@apache.org>\n\nCloses #3655 from harishreedharan/kafka-failure-fix and squashes the following commits:\n\n5e2e7ad [Hari Shreedharan] [SPARK-4704][STREAMING] Reliable Kafka Receiver can lose data if the block generator fails to store data.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/f0500f9fa378d81e4b4038a66a40eee15806b677"},{"sha":"0a89b156850fc5ba93160987927f249a7e633d51","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-4939] move to next locality when no pending tasks\n\nCurrently, if there are different locality in a task set, the tasks with NODE_LOCAL only get scheduled after all the PROCESS_LOCAL tasks are scheduled and timeout with spark.locality.wait.process (3 seconds by default). In local mode, the LocalScheduler will never call resourceOffer() again once it failed to get a task with same locality, then all the NODE_LOCAL tasks will be never scheduled.\n\nThis bug could be reproduced by run example python/streaming/stateful_network_wordcount.py, it will hang after finished a batch with some data.\n\nThis patch will check whether there is task for current locality level, if not, it will change to next locality level without waiting for `spark.locality.wait.process` seconds. It works for all locality levels.\n\nBecause the list of pending tasks are updated lazily, the check can be false-positive, it means it will not move to next locality level even there is no valid pending tasks, it will wait for timeout.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #3779 from davies/local_streaming and squashes the following commits:\n\n2d25fb3 [Davies Liu] Update TaskSetManager.scala\n1550668 [Davies Liu] add comment\n1c37aac [Davies Liu] address comments\n6b13824 [Davies Liu] address comments\n906f456 [Davies Liu] Merge branch 'master' of github.com:apache/spark into local_streaming\n414e79e [Davies Liu] fix bug, add logging\nff8eabb [Davies Liu] Merge branch 'master' into local_streaming\n28d1b3c [Davies Liu] check tasks\n9d0ceab [Davies Liu] Merge branch 'master' of github.com:apache/spark into local_streaming\n37a2804 [Davies Liu] fix tests\n49bda82 [Davies Liu] address comment\nd8fb95a [Davies Liu] move to next locality level if no more tasks\n2d6ae73 [Davies Liu] add comments\n32d363f [Davies Liu] add regression test\n7d8c5a5 [Davies Liu] jump to next locality if no pending tasks for executors","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/0a89b156850fc5ba93160987927f249a7e633d51"},{"sha":"a9f0db1fb41a6e5510d2c14f79ff086da2ecc7b0","author":{"email":"f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","name":"wangfei"},"message":"[SPARK-5591][SQL] Fix NoSuchObjectException for CTAS\n\nNow CTAS runs successfully but will throw a NoSuchObjectException.\n```\ncreate table sc as select *\nfrom (select '2011-01-11', '2011-01-11+14:18:26' from src tablesample (1 rows)\nunion all\nselect '2011-01-11', '2011-01-11+15:18:26' from src tablesample (1 rows)\nunion all\nselect '2011-01-11', '2011-01-11+16:18:26' from src tablesample (1 rows) ) s;\n```\nGet this exception:\nERROR Hive: NoSuchObjectException(message:default.sc table not found)\nat org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1560)\nat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\nat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\nat java.lang.reflect.Method.invoke(Method.java:601)\nat org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)\nat $Proxy8.get_table(Unknown Source)\nat org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:997)\nat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\nat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\nat java.lang.reflect.Method.invoke(Method.java:601)\nat org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)\nat $Proxy9.getTable(Unknown Source)\nat org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:976)\nat org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:950)\nat org.apache.spark.sql.hive.HiveMetastoreCatalog.tableExists(HiveMetastoreCatalog.scala:152)\nat org.apache.spark.sql.hive.HiveContext$$anon$2.org$apache$spark$sql$catalyst$analysis$OverrideCatalog$$super$tableExists(HiveContext.scala:309)\nat org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.tableExists(Catalog.scala:121)\nat org.apache.spark.sql.hive.HiveContext$$anon$2.tableExists(HiveContext.scala:309)\nat org.apache.spark.sql.hive.execution.CreateTableAsSelect.run(CreateTableAsSelect.scala:63)\nat org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:53)\n\nAuthor: wangfei <wangfei1@huawei.com>\n\nCloses #4365 from scwf/ctas-exception and squashes the following commits:\n\nc7c67bc [wangfei] no used imports\nf54eb2a [wangfei] fix exception for CTAS","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/a9f0db1fb41a6e5510d2c14f79ff086da2ecc7b0"},{"sha":"b90dd39793ab0a13a4559fcfff3bb5305c92ab3b","author":{"email":"f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","name":"wangfei"},"message":"[SPARK-5587][SQL] Support change database owner\n\nSupport change database owner, here i do not add the golden files since the golden answer is related to the tmp dir path (see https://github.com/scwf/spark/commit/6331e4ac0f982caf70531defcb957be76fe093c7)\n\nAuthor: wangfei <wangfei1@huawei.com>\n\nCloses #4357 from scwf/db_owner and squashes the following commits:\n\nf761533 [wangfei] remove the alter_db_owner which have added to whitelist\n79413c6 [wangfei] Revert \"added golden files\"\n6331e4a [wangfei] added golden files\n6f7cacd [wangfei] support change database owner","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/b90dd39793ab0a13a4559fcfff3bb5305c92ab3b"},{"sha":"424cb699ee9b091eb23b86dc018a86e377ad309f","author":{"email":"ed7e07f57003844c28a87fc89315331c00ab0cdc@gmail.com","name":"kul"},"message":"[SPARK-5426][SQL] Add SparkSQL Java API helper methods.\n\nRight now the PR adds few helper methods for java apis. But the issue was opened mainly to get rid of transformations in java api like `.rdd` and `.toJavaRDD` while working with `SQLContext` or `HiveContext`.\n\nAuthor: kul <kuldeep.bora@gmail.com>\n\nCloses #4243 from kul/master and squashes the following commits:\n\n2390fba [kul] [SPARK-5426][SQL] Add SparkSQL Java API helper methods.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/424cb699ee9b091eb23b86dc018a86e377ad309f"},{"sha":"417d1118cd720b616bfd30e9c1a0bfb1838e8cbd","author":{"email":"f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","name":"wangfei"},"message":"[SPARK-5367][SQL] Support star expression in udfs\n\nA follow up for #4163: support  `select array(key, *) from src`\n\nSince  array(key, *)  will not go into this case\n```\ncase Alias(f  UnresolvedFunction(_, args), name) if containsStar(args) =>\n              val expandedArgs = args.flatMap {\n                case s: Star => s.expand(child.output, resolver)\n                case o => o :: Nil\n              }\n```\nhere added a case to cover the corner case of array.\n\n/cc liancheng\n\nAuthor: wangfei <wangfei1@huawei.com>\nAuthor: scwf <wangfei1@huawei.com>\n\nCloses #4353 from scwf/udf-star1 and squashes the following commits:\n\n4350d17 [wangfei] minor fix\na7cd191 [wangfei] minor fix\n0942fb1 [wangfei] follow up: support select array(key, *) from src\n6ae00db [wangfei] also fix problem with array\nda1da09 [scwf] minor fix\nf87b5f9 [scwf] added test case\n587bf7e [wangfei] compile fix\neb93c16 [wangfei] fix star resolve issue in udf","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/417d1118cd720b616bfd30e9c1a0bfb1838e8cbd"},{"sha":"b73d5fff7208f9d09adde92c241c70095983bc09","author":{"email":"f4fb2fdc15585ce98eef9c03d7a86d94a45d60d0@126.com","name":"OopsOutOfMemory"},"message":"[SQL][Hiveconsole] Bring hive console code up to date and update README.md\n\nAdd `import org.apache.spark.sql.Dsl._` to make DSL query works.\nSince queryExecution is not avaliable in DataFrame, so remove it.\n\nAuthor: OopsOutOfMemory <victorshengli@126.com>\nAuthor: Sheng, Li <OopsOutOfMemory@users.noreply.github.com>\n\nCloses #4330 from OopsOutOfMemory/hiveconsole and squashes the following commits:\n\n46eb790 [Sheng, Li] Update SparkBuild.scala\nd23ee9f [OopsOutOfMemory] minor\nd4dd593 [OopsOutOfMemory] refine hive console","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/b73d5fff7208f9d09adde92c241c70095983bc09"},{"sha":"0d81645f776aeca6a48322da392d4fac18431556","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SQL] Correct the default size of TimestampType and expose NumericType\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #4314 from yhuai/minor and squashes the following commits:\n\nd3870a7 [Yin Huai] Update test.\n6e4b0c0 [Yin Huai] Two minor changes.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/0d81645f776aeca6a48322da392d4fac18431556"},{"sha":"548c9c2b2a1612f5e4342de322b72b9e78140618","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SQL] Use HiveContext's sessionState in HiveMetastoreCatalog.hiveDefaultTableFilePath\n\n`client.getDatabaseCurrent` uses SessionState's local variable which can be an issue.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #4355 from yhuai/defaultTablePath and squashes the following commits:\n\n84a29e5 [Yin Huai] Use HiveContext's sessionState instead of using SessionState's thread local variable.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/548c9c2b2a1612f5e4342de322b72b9e78140618"},{"sha":"e0490e271d078aa55d7c7583e2ba80337ed1b0c4","author":{"email":"c135ccafd9da216e05a17f80c5af6fcd4e4020ba@asiainfo.com","name":"guowei2"},"message":"[SPARK-5118][SQL] Fix: create table test stored as parquet as select ..\n\nAuthor: guowei2 <guowei2@asiainfo.com>\n\nCloses #3921 from guowei2/SPARK-5118 and squashes the following commits:\n\nb1ba3be [guowei2] add table file check in test case\n9da56f8 [guowei2] test case only run in Shim13\n112a0b6 [guowei2] add test case\n187c7d8 [guowei2] Fix: create table test stored as parquet as select ..","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/e0490e271d078aa55d7c7583e2ba80337ed1b0c4"},{"sha":"dc101b0e4e23dffddbc2f70d14a19fae5d87a328","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-5577] Python udf for DataFrame\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #4351 from davies/python_udf and squashes the following commits:\n\nd250692 [Davies Liu] fix conflict\n34234d4 [Davies Liu] Merge branch 'master' of github.com:apache/spark into python_udf\n440f769 [Davies Liu] address comments\nf0a3121 [Davies Liu] track life cycle of broadcast\nf99b2e1 [Davies Liu] address comments\n462b334 [Davies Liu] Merge branch 'master' of github.com:apache/spark into python_udf\n7bccc3b [Davies Liu] python udf\n58dee20 [Davies Liu] clean up","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/dc101b0e4e23dffddbc2f70d14a19fae5d87a328"},{"sha":"9a7ce70eabc0ccaa036e142fc97bf0d37faa0b63","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-5411] Allow SparkListeners to be specified in SparkConf and loaded when creating SparkContext\n\nThis patch introduces a new configuration option, `spark.extraListeners`, that allows SparkListeners to be specified in SparkConf and registered before the SparkContext is initialized.  From the configuration documentation:\n\n> A comma-separated list of classes that implement SparkListener; when initializing SparkContext, instances of these classes will be created and registered with Spark's listener bus. If a class has a single-argument constructor that accepts a SparkConf, that constructor will be called; otherwise, a zero-argument constructor will be called. If no valid constructor can be found, the SparkContext creation will fail with an exception.\n\nThis motivation for this patch is to allow monitoring code to be easily injected into existing Spark programs without having to modify those programs' code.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #4111 from JoshRosen/SPARK-5190-register-sparklistener-in-sc-constructor and squashes the following commits:\n\n8370839 [Josh Rosen] Two minor fixes after merging with master\n6e0122c [Josh Rosen] Merge remote-tracking branch 'origin/master' into SPARK-5190-register-sparklistener-in-sc-constructor\n1a5b9a0 [Josh Rosen] Remove SPARK_EXTRA_LISTENERS environment variable.\n2daff9b [Josh Rosen] Add a couple of explanatory comments for SPARK_EXTRA_LISTENERS.\nb9973da [Josh Rosen] Add test to ensure that conf and env var settings are merged, not overriden.\nd6f3113 [Josh Rosen] Use getConstructors() instead of try-catch to find right constructor.\nd0d276d [Josh Rosen] Move code into setupAndStartListenerBus() method\nb22b379 [Josh Rosen] Instantiate SparkListeners from classes listed in configurations.\n9c0d8f1 [Josh Rosen] Revert \"[SPARK-5190] Allow SparkListeners to be registered before SparkContext starts.\"\n217ecc0 [Josh Rosen] Revert \"Add addSparkListener to JavaSparkContext\"\n25988f3 [Josh Rosen] Add addSparkListener to JavaSparkContext\n163ba19 [Josh Rosen] [SPARK-5190] Allow SparkListeners to be registered before SparkContext starts.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/9a7ce70eabc0ccaa036e142fc97bf0d37faa0b63"},{"sha":"1fbd124b1bd6159086d8e88b139ce0817af02322","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-5605][SQL][DF] Allow using String to specify colum name in DSL aggregate functions\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #4376 from rxin/SPARK-5605 and squashes the following commits:\n\nc55f5fa [Reynold Xin] Added a Python test.\nf4b8dbb [Reynold Xin] [SPARK-5605][SQL][DF] Allow using String to specify colum name in DSL aggregate functions.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/1fbd124b1bd6159086d8e88b139ce0817af02322"}]}