{"push_id":541356666,"size":242,"distinct_size":242,"ref":"refs/heads/master","head":"c08238570c4aa53b0bed24d3304ce104e9bd65ce","before":"8817fc7fe8785d7b11138ca744f22f7e70f1f0a0","commits":[{"sha":"ab2abcb5ef925f15fa0e08d34a79b94a7b6578ef","author":{"email":"cdc781d9ee41870657a6bfbe4384e9f2f1f2a5dd@kelkoo.com","name":"Christophe Préaud"},"message":"[SPARK-4764] Ensure that files are fetched atomically\n\ntempFile is created in the same directory than targetFile, so that the\nmove from tempFile to targetFile is always atomic\n\nAuthor: Christophe Préaud <christophe.preaud@kelkoo.com>\n\nCloses #2855 from preaudc/master and squashes the following commits:\n\n9ba89ca [Christophe Préaud] Ensure that files are fetched atomically\n54419ae [Christophe Préaud] Merge remote-tracking branch 'upstream/master'\nc6a5590 [Christophe Préaud] Revert commit 8ea871f8130b2490f1bad7374a819bf56f0ccbbd\n7456a33 [Christophe Préaud] Merge remote-tracking branch 'upstream/master'\n8ea871f [Christophe Préaud] Ensure that files are fetched atomically","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/ab2abcb5ef925f15fa0e08d34a79b94a7b6578ef"},{"sha":"d6a972b3e4dc35a2d95df47d256462b325f4bda6","author":{"email":"e1a8399e36effe75a124102eb2a2bdfc82e0aab4@cloudera.com","name":"Kostas Sakellis"},"message":"[SPARK-4774] [SQL] Makes HiveFromSpark more portable\n\nHiveFromSpark read the kv1.txt file from SPARK_HOME/examples/src/main/resources/kv1.txt which assumed\nyou had a source tree checked out. Now we copy the kv1.txt file to a temporary file and delete it when\nthe jvm shuts down. This allows us to run this example outside of a spark source tree.\n\nAuthor: Kostas Sakellis <kostas@cloudera.com>\n\nCloses #3628 from ksakellis/kostas-spark-4774 and squashes the following commits:\n\n6770f83 [Kostas Sakellis] [SPARK-4774] [SQL] Makes HiveFromSpark more portable","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/d6a972b3e4dc35a2d95df47d256462b325f4bda6"},{"sha":"65f929d5b3a50a73cd6397bd4b72c3e7d94c99d7","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-4750] Dynamic allocation - synchronize kills\n\nSimple omission on my part.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #3612 from andrewor14/dynamic-allocation-synchronization and squashes the following commits:\n\n1f03b60 [Andrew Or] Synchronize kills","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/65f929d5b3a50a73cd6397bd4b72c3e7d94c99d7"},{"sha":"e829bfa1ab9b68f44c489d26efb042f793fd9362","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"SPARK-3926 [CORE] Reopened: result of JavaRDD collectAsMap() is not serializable\n\nMy original 'fix' didn't fix at all. Now, there's a unit test to check whether it works. Of the two options to really fix it -- copy the `Map` to a `java.util.HashMap`, or copy and modify Scala's implementation in `Wrappers.MapWrapper`, I went with the latter.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #3587 from srowen/SPARK-3926 and squashes the following commits:\n\n8586bb9 [Sean Owen] Remove unneeded no-arg constructor, and add additional note about copied code in LICENSE\n7bb0e66 [Sean Owen] Make SerializableMapWrapper actually serialize, and add unit test","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/e829bfa1ab9b68f44c489d26efb042f793fd9362"},{"sha":"cda94d15ea2a70ed3f0651ba2766b1e2f80308c1","author":{"email":"5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","name":"Sandy Ryza"},"message":"SPARK-4770. [DOC] [YARN] spark.scheduler.minRegisteredResourcesRatio doc...\n\n...umented default is incorrect for YARN\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #3624 from sryza/sandy-spark-4770 and squashes the following commits:\n\nbd81a3a [Sandy Ryza] SPARK-4770. [DOC] [YARN] spark.scheduler.minRegisteredResourcesRatio documented default is incorrect for YARN","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/cda94d15ea2a70ed3f0651ba2766b1e2f80308c1"},{"sha":"944384363d390a133529c08a1d0ac70aa8e778b5","author":{"email":"5d1ed0070fb6b6d5f66c1cf6a3d0bb37cf6f15d1@huawei.com","name":"Jacky Li"},"message":"[SQL] remove unnecessary import in spark-sql\n\nAuthor: Jacky Li <jacky.likun@huawei.com>\n\nCloses #3630 from jackylk/remove and squashes the following commits:\n\n150e7e0 [Jacky Li] remove unnecessary import","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/944384363d390a133529c08a1d0ac70aa8e778b5"},{"sha":"51b1fe1426ffecac6c4644523633ea1562ff9a4e","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-4769] [SQL] CTAS does not work when reading from temporary tables\n\nThis is the code refactor and follow ups for #2570\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #3336 from chenghao-intel/createtbl and squashes the following commits:\n\n3563142 [Cheng Hao] remove the unused variable\ne215187 [Cheng Hao] eliminate the compiling warning\n4f97f14 [Cheng Hao] fix bug in unittest\n5d58812 [Cheng Hao] revert the API changes\nb85b620 [Cheng Hao] fix the regression of temp tabl not found in CTAS","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/51b1fe1426ffecac6c4644523633ea1562ff9a4e"},{"sha":"bcb5cdad614d4fce43725dfec3ce88172d2f8c11","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-3154][STREAMING] Replace ConcurrentHashMap with mutable.HashMap and remove @volatile from 'stopped'\n\nSince `sequenceNumberToProcessor` and `stopped` are both protected by the lock `sequenceNumberToProcessor`, `ConcurrentHashMap` and `volatile` is unnecessary. So this PR updated them accordingly.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #3634 from zsxwing/SPARK-3154 and squashes the following commits:\n\n0d087ac [zsxwing] Replace ConcurrentHashMap with mutable.HashMap and remove @volatile from 'stopped'","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/bcb5cdad614d4fce43725dfec3ce88172d2f8c11"},{"sha":"383c5555c9f26c080bc9e3a463aab21dd5b3797f","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-4785][SQL] Initilize Hive UDFs on the driver and serialize them with a wrapper\n\nDifferent from Hive 0.12.0, in Hive 0.13.1 UDF/UDAF/UDTF (aka Hive function) objects should only be initialized once on the driver side and then serialized to executors. However, not all function objects are serializable (e.g. GenericUDF doesn't implement Serializable). Hive 0.13.1 solves this issue with Kryo or XML serializer. Several utility ser/de methods are provided in class o.a.h.h.q.e.Utilities for this purpose. In this PR we chose Kryo for efficiency. The Kryo serializer used here is created in Hive. Spark Kryo serializer wasn't used because there's no available SparkConf instance.\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #3640 from chenghao-intel/udf_serde and squashes the following commits:\n\n8e13756 [Cheng Hao] Update the comment\n74466a3 [Cheng Hao] refactor as feedbacks\n396c0e1 [Cheng Hao] avoid Simple UDF to be serialized\ne9c3212 [Cheng Hao] update the comment\n19cbd46 [Cheng Hao] support udf instance ser/de after initialization","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/383c5555c9f26c080bc9e3a463aab21dd5b3797f"},{"sha":"912563aa3553afc0871d5b5858f533aa39cb99e5","author":{"email":"5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","name":"Sandy Ryza"},"message":"SPARK-4338. [YARN] Ditch yarn-alpha.\n\nSorry if this is a little premature with 1.2 still not out the door, but it will make other work like SPARK-4136 and SPARK-2089 a lot easier.\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #3215 from sryza/sandy-spark-4338 and squashes the following commits:\n\n1c5ac08 [Sandy Ryza] Update building Spark docs and remove unnecessary newline\n9c1421c [Sandy Ryza] SPARK-4338. Ditch yarn-alpha.","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/912563aa3553afc0871d5b5858f533aa39cb99e5"},{"sha":"61f1a7022767e64ab092aa91c0c5aa1b2fdbef7c","author":{"email":"054cbd1708f02aafe84688c98217f6c4f69e823c@gmail.com","name":"jbencook"},"message":"[SPARK-874] adding a --wait flag\n\nThis PR adds a --wait flag to the `./sbin/stop-all.sh` script.\n\nAuthor: jbencook <jbenjamincook@gmail.com>\n\nCloses #3567 from jbencook/master and squashes the following commits:\n\nd05c5bb [jbencook] [SPARK-874] adding a --wait flag","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/61f1a7022767e64ab092aa91c0c5aa1b2fdbef7c"},{"sha":"b31074466a83d3d1387fc1e4337dfab9e164fc04","author":{"email":"5cc7173406045a3242d3595588aa610a46f4e47d@asiainfo.com","name":"maji2014"},"message":" [SPARK-4691][shuffle] Restructure a few lines in shuffle code\n\nIn HashShuffleReader.scala and HashShuffleWriter.scala, no need to judge \"dep.aggregator.isEmpty\" again as this is judged by \"dep.aggregator.isDefined\"\n\nIn SortShuffleWriter.scala, \"dep.aggregator.isEmpty\"  is better than \"!dep.aggregator.isDefined\" ?\n\nAuthor: maji2014 <maji3@asiainfo.com>\n\nCloses #3553 from maji2014/spark-4691 and squashes the following commits:\n\nbf7b14d [maji2014] change a elegant way for SortShuffleWriter.scala\n10d0cf0 [maji2014] change a elegant way\nd8f52dc [maji2014] code optimization for judgement","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/b31074466a83d3d1387fc1e4337dfab9e164fc04"},{"sha":"1f5110630c1abb13a357b463c805a39772923b82","author":{"email":"284a00ea9901227c42561196ff8576aed3ac5631@gmail.com","name":"Kay Ousterhout"},"message":"[SPARK-4765] Make GC time always shown in UI.\n\nThis commit removes the GC time for each task from the set of\noptional, additional metrics, and instead always shows it for\neach task.\n\ncc pwendell\n\nAuthor: Kay Ousterhout <kayousterhout@gmail.com>\n\nCloses #3622 from kayousterhout/gc_time and squashes the following commits:\n\n15ac242 [Kay Ousterhout] Make TaskDetailsClassNames private[spark]\ne71d893 [Kay Ousterhout] [SPARK-4765] Make GC time always shown in UI.","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/1f5110630c1abb13a357b463c805a39772923b82"},{"sha":"30dca924df0efbdc1b638fa7c705fe8743570783","author":{"email":"f4cfeb6da63cd4b62a6340d67bbf18015ad432f2@xiaomi.com","name":"hushan[胡珊]"},"message":"[SPARK-4714] BlockManager.dropFromMemory() should check whether block has been removed after synchronizing on BlockInfo instance.\n\nAfter synchronizing on the `info` lock in the `removeBlock`/`dropOldBlocks`/`dropFromMemory` methods in BlockManager, the block that `info` represented may have already removed.\n\nThe three methods have the same logic to get the `info` lock:\n```\n   info = blockInfo.get(id)\n   if (info != null) {\n     info.synchronized {\n       // do something\n     }\n   }\n```\n\nSo, there is chance that when a thread enters the `info.synchronized` block, `info` has already been removed from the `blockInfo` map by some other thread who entered `info.synchronized` first.\n\nThe `removeBlock` and `dropOldBlocks` methods are idempotent, so it's safe for them to run on blocks that have already been removed.\nBut in `dropFromMemory` it may be problematic since it may drop block data which already removed into the diskstore, and this calls data store operations that are not designed to handle missing blocks.\n\nThis patch fixes this issue by adding a check to `dropFromMemory` to test whether blocks have been removed by a racing thread.\n\nAuthor: hushan[胡珊] <hushan@xiaomi.com>\n\nCloses #3574 from suyanNone/refine-block-concurrency and squashes the following commits:\n\nedb989d [hushan[胡珊]] Refine code style and comments position\n55fa4ba [hushan[胡珊]] refine code\ne57e270 [hushan[胡珊]] add check info is already remove or not while having gotten info.syn","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/30dca924df0efbdc1b638fa7c705fe8743570783"},{"sha":"5e4c06f8e54265a4024857f5978ec54c936aeea2","author":{"email":"5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","name":"Sandy Ryza"},"message":"SPARK-4567. Make SparkJobInfo and SparkStageInfo serializable\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #3426 from sryza/sandy-spark-4567 and squashes the following commits:\n\ncb4b8d2 [Sandy Ryza] SPARK-4567. Make SparkJobInfo and SparkStageInfo serializable","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/5e4c06f8e54265a4024857f5978ec54c936aeea2"},{"sha":"d8f84f26e388055ca7459810e001d05ab60af15b","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"SPARK-4805 [CORE] BlockTransferMessage.toByteArray() trips assertion\n\nAllocate enough room for type byte as well as message, to avoid tripping assertion about capacity of the buffer\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #3650 from srowen/SPARK-4805 and squashes the following commits:\n\n9e1d502 [Sean Owen] Allocate enough room for type byte as well as message, to avoid tripping assertion about capacity of the buffer","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/d8f84f26e388055ca7459810e001d05ab60af15b"},{"sha":"2b9b72682e587909a84d3ace214c22cec830eeaf","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-4740] Create multiple concurrent connections between two peer nodes in Netty.\n\nIt's been reported that when the number of disks is large and the number of nodes is small, Netty network throughput is low compared with NIO. We suspect the problem is that only a small number of disks are utilized to serve shuffle files at any given point, due to connection reuse. This patch adds a new config parameter to specify the number of concurrent connections between two peer nodes, default to 2.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #3625 from rxin/SPARK-4740 and squashes the following commits:\n\nad4241a [Reynold Xin] Updated javadoc.\nf33c72b [Reynold Xin] Code review feedback.\n0fefabb [Reynold Xin] Use double check in synchronization.\n41dfcb2 [Reynold Xin] Added test case.\n9076b4a [Reynold Xin] Fixed two NPEs.\n3e1306c [Reynold Xin] Minor style fix.\n4f21673 [Reynold Xin] [SPARK-4740] Create multiple concurrent connections between two peer nodes in Netty.","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/2b9b72682e587909a84d3ace214c22cec830eeaf"},{"sha":"9bd9334f588dbb44d01554f9f4ca68a153a48993","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Config updates for the new shuffle transport.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #3657 from rxin/conf-update and squashes the following commits:\n\n7370eab [Reynold Xin] Config updates for the new shuffle transport.","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/9bd9334f588dbb44d01554f9f4ca68a153a48993"},{"sha":"f79c1cfc997c1a7ddee480ca3d46f5341b69d3b7","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[Minor] Use <sup> tag for help icon in web UI page header\n\nThis small commit makes the `(?)` web UI help link into a superscript, which should address feedback that the current design makes it look like an error occurred or like information is missing.\n\nBefore:\n\n![image](https://cloud.githubusercontent.com/assets/50748/5370611/a3ed0034-7fd9-11e4-870f-05bd9faad5b9.png)\n\nAfter:\n\n![image](https://cloud.githubusercontent.com/assets/50748/5370602/6c5ca8d6-7fd9-11e4-8d1a-568d71290aa7.png)\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #3659 from JoshRosen/webui-help-sup and squashes the following commits:\n\nbd72899 [Josh Rosen] Use <sup> tag for help icon in web UI page header.","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/f79c1cfc997c1a7ddee480ca3d46f5341b69d3b7"},{"sha":"94b377f94487109a1cc3e07dd230b1df7a96e28d","author":{"email":"73279e3fb6c013b10d08fa3ac4497d0f034eab9f@oculusinfo.com","name":"Nathan Kronenfeld"},"message":"[SPARK-4772] Clear local copies of accumulators as soon as we're done with them\n\nAccumulators keep thread-local copies of themselves.  These copies were only cleared at the beginning of a task.  This meant that (a) the memory they used was tied up until the next task ran on that thread, and (b) if a thread died, the memory it had used for accumulators was locked up forever on that worker.\n\nThis PR clears the thread-local copies of accumulators at the end of each task, in the tasks finally block, to make sure they are cleaned up between tasks.  It also stores them in a ThreadLocal object, so that if, for some reason, the thread dies, any memory they are using at the time should be freed up.\n\nAuthor: Nathan Kronenfeld <nkronenfeld@oculusinfo.com>\n\nCloses #3570 from nkronenfeld/Accumulator-Improvements and squashes the following commits:\n\na581f3f [Nathan Kronenfeld] Change Accumulators to private[spark] instead of adding mima exclude to get around false positive in mima tests\nb6c2180 [Nathan Kronenfeld] Include MiMa exclude as per build error instructions - this version incompatibility should be irrelevent, as it will only surface if a master is talking to a worker running a different version of spark.\n537baad [Nathan Kronenfeld] Fuller refactoring as intended, incorporating JR's suggestions for ThreadLocal localAccums, and keeping clear(), but also calling it in tasks' finally block, rather than just at the beginning of the task.\n39a82f2 [Nathan Kronenfeld] Clear local copies of accumulators as soon as we're done with them","distinct":true,"url":"https://api.github.com/repos/rnowling/spark/commits/94b377f94487109a1cc3e07dd230b1df7a96e28d"}]}