{"push_id":893930737,"size":16,"distinct_size":16,"ref":"refs/heads/branch-1.5","head":"3868ab644cc87ce68ba1605f6da65c5e951ce412","before":"e9ae1fda9e6009cf95f9a98ba130297126155e06","commits":[{"sha":"27b5f31a08737969b69a43536acc04cb882a7bdc","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-11836][SQL] use existing SQLContext for udf/cast (1.5 branch)\n\nudf/cast should use existing SQLContext.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #9915 from davies/create_1.5.","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/27b5f31a08737969b69a43536acc04cb882a7bdc"},{"sha":"4139a4ed1f024b1a88e4efe613acf45175f82318","author":{"email":"47f947204c0a430d1f3d77ee35599c230fda5709@gmail.com","name":"Mark Hamstra"},"message":"[SPARK-10666][SPARK-6880][CORE] Use properties from ActiveJob associated with a Stage\n\nThis issue was addressed in https://github.com/apache/spark/pull/5494, but the fix in that PR, while safe in the sense that it will prevent the SparkContext from shutting down, misses the actual bug.  The intent of `submitMissingTasks` should be understood as \"submit the Tasks that are missing for the Stage, and run them as part of the ActiveJob identified by jobId\".  Because of a long-standing bug, the `jobId` parameter was never being used.  Instead, we were trying to use the jobId with which the Stage was created -- which may no longer exist as an ActiveJob, hence the crash reported in SPARK-6880.\n\nThe correct fix is to use the ActiveJob specified by the supplied jobId parameter, which is guaranteed to exist at the call sites of submitMissingTasks.\n\nThis fix should be applied to all maintenance branches, since it has existed since 1.0.\n\nkayousterhout pankajarora12\n\nAuthor: Mark Hamstra <markhamstra@gmail.com>\nAuthor: Imran Rashid <irashid@cloudera.com>\n\nCloses #6291 from markhamstra/SPARK-6880.\n\n(cherry picked from commit 0a5aef753e70e93d7e56054f354a52e4d4e18932)\nSigned-off-by: Imran Rashid <irashid@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/4139a4ed1f024b1a88e4efe613acf45175f82318"},{"sha":"b1fcefca6b257f5bb1be85fce2b4a97d4271e0aa","author":{"email":"484bb3ee2eae99d0014a3b8fd10f5274ab6daf82@huawei.com","name":"Zhongshuai Pei"},"message":"[SPARK-11974][CORE] Not all the temp dirs had been deleted when the JVM exits\n\ndeleting the temp dir like that\n\n```\n\nscala> import scala.collection.mutable\nimport scala.collection.mutable\n\nscala> val a = mutable.Set(1,2,3,4,7,0,8,98,9)\na: scala.collection.mutable.Set[Int] = Set(0, 9, 1, 2, 3, 7, 4, 8, 98)\n\nscala> a.foreach(x => {a.remove(x) })\n\nscala> a.foreach(println(_))\n98\n```\n\nYou may not modify a collection while traversing or iterating over it.This can not delete all element of the collection\n\nAuthor: Zhongshuai Pei <peizhongshuai@huawei.com>\n\nCloses #9951 from DoingDone9/Bug_RemainDir.\n\n(cherry picked from commit 6b781576a15d8d5c5fbed8bef1c5bda95b3d44ac)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/b1fcefca6b257f5bb1be85fce2b4a97d4271e0aa"},{"sha":"7900d192e8adf501fbed0d0704d60d2c0e63a764","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-11999][CORE] Fix the issue that ThreadUtils.newDaemonCachedThreadPool doesn't cache any task\n\nIn the previous codes, `newDaemonCachedThreadPool` uses `SynchronousQueue`, which is wrong. `SynchronousQueue` is an empty queue that cannot cache any task. This patch uses `LinkedBlockingQueue` to fix it along with other fixes to make sure `newDaemonCachedThreadPool` can use at most `maxThreadNumber` threads, and after that, cache tasks to `LinkedBlockingQueue`.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #9978 from zsxwing/cached-threadpool.\n\n(cherry picked from commit d3ef693325f91a1ed340c9756c81244a80398eb2)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/7900d192e8adf501fbed0d0704d60d2c0e63a764"},{"sha":"9b62161715950c1b76385cf2c2e2e46213fba437","author":{"email":"dfc272d1e78652e4de5b4e76a0545cef11a46a5c@gmail.com","name":"Teng Qiu"},"message":"[SPARK-12053][CORE] EventLoggingListener.getLogPath needs 4 parameters\n\n```EventLoggingListener.getLogPath``` needs 4 input arguments:\nhttps://github.com/apache/spark/blob/v1.6.0-preview2/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala#L276-L280\n\nthe 3rd parameter should be appAttemptId, 4th parameter is codec...\n\nAuthor: Teng Qiu <teng.qiu@gmail.com>\n\nCloses #10044 from chutium/SPARK-12053.\n\n(cherry picked from commit a8ceec5e8c1572dd3d74783c06c78b7ca0b8a7ce)\nSigned-off-by: Kousuke Saruta <sarutak@oss.nttdata.co.jp>","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/9b62161715950c1b76385cf2c2e2e46213fba437"},{"sha":"d78f1bc45ff04637761bcfcc565ce40fb587a5a9","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-12049][CORE] User JVM shutdown hook can cause deadlock at shutdown\n\nAvoid potential deadlock with a user app's shutdown hook thread by more narrowly synchronizing access to 'hooks'\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #10042 from srowen/SPARK-12049.\n\n(cherry picked from commit 96bf468c7860be317c20ccacf259910968d2dc83)\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/d78f1bc45ff04637761bcfcc565ce40fb587a5a9"},{"sha":"80dac0b07dd37064255f13874ebae5f75e371e9c","author":{"email":"032edf494edf376e1e59f71481ba69cbe7ab4358@gmail.com","name":"Alexander Pivovarov"},"message":"Set SPARK_EC2_VERSION to 1.5.2\n\nAuthor: Alexander Pivovarov <apivovarov@gmail.com>\n\nCloses #10064 from apivovarov/patch-1.","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/80dac0b07dd37064255f13874ebae5f75e371e9c"},{"sha":"f28399e1ab6fd3a829d05ffbc637aa2c86cffdf2","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@databricks.com","name":"Nong Li"},"message":"[SPARK-11328][SQL] Improve error message when hitting this issue\n\nThe issue is that the output commiter is not idempotent and retry attempts will\nfail because the output file already exists. It is not safe to clean up the file\nas this output committer is by design not retryable. Currently, the job fails\nwith a confusing file exists error. This patch is a stop gap to tell the user\nto look at the top of the error log for the proper message.\n\nThis is difficult to test locally as Spark is hardcoded not to retry. Manually\nverified by upping the retry attempts.\n\nAuthor: Nong Li <nong@databricks.com>\nAuthor: Nong Li <nongli@gmail.com>\n\nCloses #10080 from nongli/spark-11328.\n\n(cherry picked from commit 47a0abc343550c855e679de12983f43e6fcc0171)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/f28399e1ab6fd3a829d05ffbc637aa2c86cffdf2"},{"sha":"fc3fb84636ba5f84220e1f46036841f1cb61f1d0","author":{"email":"b1011540267ee7fe5a02243ad09c007ca1860809@oc0558782468.ibm.com","name":"Huaxin Gao"},"message":"[SPARK-11788][SQL] surround timestamp/date value with quotes in JDBC data source\n\nWhen query the Timestamp or Date column like the following\nval filtered = jdbcdf.where($\"TIMESTAMP_COLUMN\" >= beg && $\"TIMESTAMP_COLUMN\" < end)\nThe generated SQL query is \"TIMESTAMP_COLUMN >= 2015-01-01 00:00:00.0\"\nIt should have quote around the Timestamp/Date value such as \"TIMESTAMP_COLUMN >= '2015-01-01 00:00:00.0'\"\n\nAuthor: Huaxin Gao <huaxing@oc0558782468.ibm.com>\n\nCloses #9872 from huaxingao/spark-11788.\n\n(cherry picked from commit 5a8b5fdd6ffa58f015cdadf3f2c6df78e0a388ad)\nSigned-off-by: Yin Huai <yhuai@databricks.com>\n\nConflicts:\n\tsql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/fc3fb84636ba5f84220e1f46036841f1cb61f1d0"},{"sha":"7460e430929473152230d70964a04a7ff834066c","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@databricks.com","name":"Nong Li"},"message":"[SPARK-12030] Fix Platform.copyMemory to handle overlapping regions.\n\nThis bug was exposed as memory corruption in Timsort which uses copyMemory to copy\nlarge regions that can overlap. The prior implementation did not handle this case\nhalf the time and always copied forward, resulting in the data being corrupt.\n\nAuthor: Nong Li <nong@databricks.com>\n\nCloses #10068 from nongli/spark-12030.\n\n(cherry picked from commit 2cef1cdfbb5393270ae83179b6a4e50c3cbf9e93)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/7460e430929473152230d70964a04a7ff834066c"},{"sha":"4f07a590c5d7b9e187f446c077357b00df93ee27","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-11352][SQL][BRANCH-1.5] Escape */ in the generated comments.\n\nhttps://issues.apache.org/jira/browse/SPARK-11352\n\nThis one backports https://github.com/apache/spark/pull/10072 to branch 1.5.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #10084 from yhuai/SPARK-11352-branch-1.5.","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/4f07a590c5d7b9e187f446c077357b00df93ee27"},{"sha":"0d57a4ae10f4ec40386194bc3c8e27f32da09d4d","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-12087][STREAMING] Create new JobConf for every batch in saveAsHadoopFiles\n\nThe JobConf object created in `DStream.saveAsHadoopFiles` is used concurrently in multiple places:\n* The JobConf is updated by `RDD.saveAsHadoopFile()` before the job is launched\n* The JobConf is serialized as part of the DStream checkpoints.\nThese concurrent accesses (updating in one thread, while the another thread is serializing it) can lead to concurrentModidicationException in the underlying Java hashmap using in the internal Hadoop Configuration object.\n\nThe solution is to create a new JobConf in every batch, that is updated by `RDD.saveAsHadoopFile()`, while the checkpointing serializes the original JobConf.\n\nTests to be added in #9988 will fail reliably without this patch. Keeping this patch really small to make sure that it can be added to previous branches.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #10088 from tdas/SPARK-12087.\n\n(cherry picked from commit 8a75a3049539eeef04c0db51736e97070c162b46)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/0d57a4ae10f4ec40386194bc3c8e27f32da09d4d"},{"sha":"ed7264ba20d178be45dad3f50e9c73ce2f9148dc","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12090] [PYSPARK] consider shuffle in coalesce()\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10090 from davies/fix_coalesce.\n\n(cherry picked from commit 4375eb3f48fc7ae90caf6c21a0d3ab0b66bf4efa)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/ed7264ba20d178be45dad3f50e9c73ce2f9148dc"},{"sha":"8bbb3cdd140cf0ddf55ac4cdafb3b59024bf20fe","author":{"email":"0cbb753a7611206246cbe14e9d56f956ba7f6dea@condor.rhaag.ip","name":"gcc"},"message":"[SPARK-12048][SQL] Prevent to close JDBC resources twice\n\nAuthor: gcc <spark-src@condor.rhaag.ip>\n\nCloses #10101 from rh99/master.\n\n(cherry picked from commit 04b6799932707f0a4aa4da0f2fc838bdb29794ce)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/8bbb3cdd140cf0ddf55ac4cdafb3b59024bf20fe"},{"sha":"93a0510a5d623172c022dbb70c779f1653e0cd6f","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-12138][SQL] Escape \\u in the generated comments of codegen\n\nWhen \\u appears in a comment block (i.e. in /**/), code gen will break. So, in Expression and CodegenFallback, we escape \\u to \\\\u.\n\nyhuai Please review it. I did reproduce it and it works after the fix. Thanks!\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #10155 from gatorsmile/escapeU.\n\n(cherry picked from commit 49efd03bacad6060d99ed5e2fe53ba3df1d1317e)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/93a0510a5d623172c022dbb70c779f1653e0cd6f"},{"sha":"3868ab644cc87ce68ba1605f6da65c5e951ce412","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-12101][CORE] Fix thread pools that cannot cache tasks in Worker and AppClient (backport 1.5)\n\nbackport #10108 to branch 1.5\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #10135 from zsxwing/fix-threadpool-1.5.","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/3868ab644cc87ce68ba1605f6da65c5e951ce412"}]}