{"push_id":640424896,"size":27,"distinct_size":27,"ref":"refs/heads/master","head":"bdc5c16e76c5d0bc147408353b2ba4faa8e914fc","before":"c736220dac51cf73181fdd7f621c960c4e7bf0c2","commits":[{"sha":"8136810dfad12008ac300116df7bc8448740f1ae","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-6490][Core] Add spark.rpc.* and deprecate spark.akka.*\n\nDeprecated `spark.akka.num.retries`, `spark.akka.retry.wait`, `spark.akka.askTimeout`,  `spark.akka.lookupTimeout`, and added `spark.rpc.num.retries`, `spark.rpc.retry.wait`, `spark.rpc.askTimeout`, `spark.rpc.lookupTimeout`.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #5595 from zsxwing/SPARK-6490 and squashes the following commits:\n\ne0d80a9 [zsxwing] Use getTimeAsMs and getTimeAsSeconds and other minor fixes\n31dbe69 [zsxwing] Add spark.rpc.* and deprecate spark.akka.*","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/8136810dfad12008ac300116df7bc8448740f1ae"},{"sha":"ab9128fb7ec7ca479dc91e7cc7c775e8d071eafa","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-6949] [SQL] [PySpark] Support Date/Timestamp in Column expression\n\nThis PR enable auto_convert in JavaGateway, then we could register a converter for a given types, for example, date and datetime.\n\nThere are two bugs related to auto_convert, see [1] and [2], we workaround it in this PR.\n\n[1]  https://github.com/bartdag/py4j/issues/160\n[2] https://github.com/bartdag/py4j/issues/161\n\ncc rxin JoshRosen\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #5570 from davies/py4j_date and squashes the following commits:\n\neb4fa53 [Davies Liu] fix tests in python 3\nd17d634 [Davies Liu] rollback changes in mllib\n2e7566d [Davies Liu] convert tuple into ArrayList\nceb3779 [Davies Liu] Update rdd.py\n3c373f3 [Davies Liu] support date and datetime by auto_convert\ncb094ff [Davies Liu] enable auto convert","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/ab9128fb7ec7ca479dc91e7cc7c775e8d071eafa"},{"sha":"1f2f723b0daacbb9e70ec42c19a84470af1d7765","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-5990] [MLLIB] Model import/export for IsotonicRegression\n\nModel import/export for IsotonicRegression\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #5270 from yanboliang/spark-5990 and squashes the following commits:\n\n872028d [Yanbo Liang] fix code style\nf80ec1b [Yanbo Liang] address comments\n49600cc [Yanbo Liang] address comments\n429ff7d [Yanbo Liang] store each interval as a record\n2b2f5a1 [Yanbo Liang] Model import/export for IsotonicRegression","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/1f2f723b0daacbb9e70ec42c19a84470af1d7765"},{"sha":"5fea3e5c36450658d8b767dd3c06dac2251a0e0c","author":{"email":"05cc54f0b2dee1599e14a641f80cd8e7d4fe1230@nike.com","name":"David McGuire"},"message":"[SPARK-6985][streaming] Receiver maxRate over 1000 causes a StackOverflowError\n\nA simple truncation in integer division (on rates over 1000 messages / second) causes the existing implementation to sleep for 0 milliseconds, then call itself recursively; this causes what is essentially an infinite recursion, since the base case of the calculated amount of time having elapsed can't be reached before available stack space is exhausted. A fix to this truncation error is included in this patch.\n\nHowever, even with the defect patched, the accuracy of the existing implementation is abysmal (the error bounds of the original test were effectively [-30%, +10%], although this fact was obscured by hard-coded error margins); as such, when the error bounds were tightened down to [-5%, +5%], the existing implementation failed to meet the new, tightened, requirements. Therefore, an industry-vetted solution (from Guava) was used to get the adapted tests to pass.\n\nAuthor: David McGuire <david.mcguire2@nike.com>\n\nCloses #5559 from dmcguire81/master and squashes the following commits:\n\nd29d2e0 [David McGuire] Back out to +/-5% error margins, for flexibility in timing\n8be6934 [David McGuire] Fix spacing per code review\n90e98b9 [David McGuire] Address scalastyle errors\n29011bd [David McGuire] Further ratchet down the error margins\nb33b796 [David McGuire] Eliminate dependency on even distribution by BlockGenerator\n8f2934b [David McGuire] Remove arbitrary thread timing / cooperation code\n70ee310 [David McGuire] Use Thread.yield(), since Thread.sleep(0) is system-dependent\n82ee46d [David McGuire] Replace guard clause with nested conditional\n2794717 [David McGuire] Replace the RateLimiter with the Guava implementation\n38f3ca8 [David McGuire] Ratchet down the error rate to +/- 5%; tests fail\n24b1bc0 [David McGuire] Fix truncation in integer division causing infinite recursion\nd6e1079 [David McGuire] Stack overflow error in RateLimiter on rates over 1000/s","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/5fea3e5c36450658d8b767dd3c06dac2251a0e0c"},{"sha":"c035c0f2d72f2a303b86fe0037ec43d756fff060","author":{"email":"284a00ea9901227c42561196ff8576aed3ac5631@gmail.com","name":"Kay Ousterhout"},"message":"[SPARK-5360] [SPARK-6606] Eliminate duplicate objects in serialized CoGroupedRDD\n\nCoGroupPartition, part of CoGroupedRDD, includes references to each RDD that the CoGroupedRDD narrowly depends on, and a reference to the ShuffleHandle. The partition is serialized separately from the RDD, so when the RDD and partition arrive on the worker, the references in the partition and in the RDD no longer point to the same object.\n\nThis is a relatively minor performance issue (the closure can be 2x larger than it needs to be because the rdds and partitions are serialized twice; see numbers below) but is more annoying as a developer issue (this is where I ran into): if any state is stored in the RDD or ShuffleHandle on the worker side, subtle bugs can appear due to the fact that the references to the RDD / ShuffleHandle in the RDD and in the partition point to separate objects. I'm not sure if this is enough of a potential future problem to fix this old and central part of the code, so hoping to get input from others here.\n\nI did some simple experiments to see how much this effects closure size. For this example:\n$ val a = sc.parallelize(1 to 10).map((_, 1))\n$ val b = sc.parallelize(1 to 2).map(x => (x, 2*x))\n$ a.cogroup(b).collect()\nthe closure was 1902 bytes with current Spark, and 1129 bytes after my change. The difference comes from eliminating duplicate serialization of the shuffle handle.\n\nFor this example:\n$ val sortedA = a.sortByKey()\n$ val sortedB = b.sortByKey()\n$ sortedA.cogroup(sortedB).collect()\nthe closure was 3491 bytes with current Spark, and 1333 bytes after my change. Here, the difference comes from eliminating duplicate serialization of the two RDDs for the narrow dependencies.\n\nThe ShuffleHandle includes the ShuffleDependency, so this difference will get larger if a ShuffleDependency includes a serializer, a key ordering, or an aggregator (all set to None by default). It would also get bigger for a big RDD -- although I can't think of any examples where the RDD object gets large.  The difference is not affected by the size of the function the user specifies, which (based on my understanding) is typically the source of large task closures.\n\nAuthor: Kay Ousterhout <kayousterhout@gmail.com>\n\nCloses #4145 from kayousterhout/SPARK-5360 and squashes the following commits:\n\n85156c3 [Kay Ousterhout] Better comment the narrowDeps parameter\ncff0209 [Kay Ousterhout] Fixed spelling issue\n658e1af [Kay Ousterhout] [SPARK-5360] Eliminate duplicate objects in serialized CoGroupedRDD","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/c035c0f2d72f2a303b86fe0037ec43d756fff060"},{"sha":"c25ca7c5a1f2a4f88f40b0c5cdbfa927c186cfa8","author":{"email":"27272cb5e2aa23ca8534abd304264d1be4536777@gmail.com","name":"emres"},"message":"SPARK-3276 Added a new configuration spark.streaming.minRememberDuration\n\nSPARK-3276 Added a new configuration parameter ``spark.streaming.minRememberDuration``, with a default value of 1 minute.\n\nSo that when a Spark Streaming application is started, an arbitrary number of minutes can be taken as threshold for remembering.\n\nAuthor: emres <emre.sevinc@gmail.com>\n\nCloses #5438 from emres/SPARK-3276 and squashes the following commits:\n\n766f938 [emres] SPARK-3276 Switched to using newly added getTimeAsSeconds method.\naffee1d [emres] SPARK-3276 Changed the property name and variable name for minRememberDuration\nc9d58ca [emres] SPARK-3276 Minor code re-formatting.\n1c53ba9 [emres] SPARK-3276 Started to use ssc.conf rather than ssc.sparkContext.getConf,  and also getLong method directly.\nbfe0acb [emres] SPARK-3276 Moved the minRememberDurationMin to the class\ndaccc82 [emres] SPARK-3276 Changed the property name to reflect the unit of value and reduced number of fields.\n43cc1ce [emres] SPARK-3276 Added a new configuration parameter spark.streaming.minRemember duration, with a default value of 1 minute.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/c25ca7c5a1f2a4f88f40b0c5cdbfa927c186cfa8"},{"sha":"45c47fa4176ea75886a58f5d73c44afcb29aa629","author":{"email":"8757c4a255384de9fc2175f17a84f948e7081d30@gmail.com","name":"MechCoder"},"message":"[SPARK-6845] [MLlib] [PySpark] Add isTranposed flag to DenseMatrix\n\nSince sparse matrices now support a isTransposed flag for row major data, DenseMatrices should do the same.\n\nAuthor: MechCoder <manojkumarsivaraj334@gmail.com>\n\nCloses #5455 from MechCoder/spark-6845 and squashes the following commits:\n\n525c370 [MechCoder] minor\n004a37f [MechCoder] Cast boolean to int\n151f3b6 [MechCoder] [WIP] Add isTransposed to pickle DenseMatrix\ncc0b90a [MechCoder] [SPARK-6845] Add isTranposed flag to DenseMatrix","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/45c47fa4176ea75886a58f5d73c44afcb29aa629"},{"sha":"04bf34e34f22e3d7e972fe755251774fc6a6d52e","author":{"email":"38ba40da102c3b82607690471c403766355c3e07@imaginea.com","name":"Prashant Sharma"},"message":"[SPARK-7011] Build(compilation) fails with scala 2.11 option, because a protected[sql] type is accessed in ml package.\n\n[This](https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/feature/VectorAssembler.scala#L58) is where it is used and fails compilations at.\n\nAuthor: Prashant Sharma <prashant.s@imaginea.com>\n\nCloses #5593 from ScrapCodes/SPARK-7011/build-fix and squashes the following commits:\n\ne6d57a3 [Prashant Sharma] [SPARK-7011] Build fails with scala 2.11 option, because a protected[sql] type is accessed in ml package.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/04bf34e34f22e3d7e972fe755251774fc6a6d52e"},{"sha":"2e8c6ca47df14681c1110f0736234ce76a3eca9b","author":{"email":"245b2ef566c711104e8473de992ea4e7282080c1@vinted.com","name":"vidmantas zemleris"},"message":"[SPARK-6994] Allow to fetch field values by name in sql.Row\n\nIt looked weird that up to now there was no way in Spark's Scala API to access fields of `DataFrame/sql.Row` by name, only by their index.\n\nThis tries to solve this issue.\n\nAuthor: vidmantas zemleris <vidmantas@vinted.com>\n\nCloses #5573 from vidma/features/row-with-named-fields and squashes the following commits:\n\n6145ae3 [vidmantas zemleris] [SPARK-6994][SQL] Allow to fetch field values by name on Row\n9564ebb [vidmantas zemleris] [SPARK-6994][SQL] Add fieldIndex to schema (StructType)","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/2e8c6ca47df14681c1110f0736234ce76a3eca9b"},{"sha":"03fd92167107f1d061c1a7ef216468b508546ac7","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SQL][minor] make it more clear that we only need to re-throw GetField exception for UnresolvedAttribute\n\nFor `GetField` outside `UnresolvedAttribute`, we will throw exception in `Analyzer`.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #5588 from cloud-fan/tmp and squashes the following commits:\n\n7ac74d2 [Wenchen Fan] small refactor","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/03fd92167107f1d061c1a7ef216468b508546ac7"},{"sha":"6265cba00f6141575b4be825735d77d4cea500ab","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-6969][SQL] Refresh the cached table when REFRESH TABLE is used\n\nhttps://issues.apache.org/jira/browse/SPARK-6969\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #5583 from yhuai/refreshTableRefreshDataCache and squashes the following commits:\n\n1e5142b [Yin Huai] Add todo.\n92b2498 [Yin Huai] Minor updates.\n367df92 [Yin Huai] Recache data in the command of REFRESH TABLE.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/6265cba00f6141575b4be825735d77d4cea500ab"},{"sha":"2a24bf92e6d36e876bad6a8b4e0ff12c407ebb8a","author":{"email":"29e05320a613faa71033b8bf4c133bf8ae9edea8@palantir.com","name":"Punya Biswal"},"message":"[SPARK-6996][SQL] Support map types in java beans\n\nliancheng mengxr this is similar to #5146.\n\nAuthor: Punya Biswal <pbiswal@palantir.com>\n\nCloses #5578 from punya/feature/SPARK-6996 and squashes the following commits:\n\nd56c3e0 [Punya Biswal] Fix imports\nc7e308b [Punya Biswal] Support java iterable types in POJOs\n5e00685 [Punya Biswal] Support map types in java beans","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/2a24bf92e6d36e876bad6a8b4e0ff12c407ebb8a"},{"sha":"7662ec23bb6c4d4fe4c857b6928eaed0a97d3c04","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-5817] [SQL] Fix bug of udtf with column names\n\nIt's a bug while do query like:\n```sql\nselect d from (select explode(array(1,1)) d from src limit 1) t\n```\nAnd it will throws exception like:\n```\norg.apache.spark.sql.AnalysisException: cannot resolve 'd' given input columns _c0; line 1 pos 7\nat org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)\nat org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$apply$3$$anonfun$apply$1.applyOrElse(CheckAnalysis.scala:48)\nat org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$apply$3$$anonfun$apply$1.applyOrElse(CheckAnalysis.scala:45)\nat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:250)\nat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:250)\nat org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:50)\nat org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:249)\nat org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$transformExpressionUp$1(QueryPlan.scala:103)\nat org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$2$$anonfun$apply$2.apply(QueryPlan.scala:117)\nat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)\nat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)\nat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\nat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)\nat scala.collection.TraversableLike$class.map(TraversableLike.scala:244)\nat scala.collection.AbstractTraversable.map(Traversable.scala:105)\nat org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$2.apply(QueryPlan.scala:116)\nat scala.collection.Iterator$$anon$11.next(Iterator.scala:328)\n```\n\nTo solve the bug, it requires code refactoring for UDTF\nThe major changes are about:\n* Simplifying the UDTF development, UDTF will manage the output attribute names any more, instead, the `logical.Generate` will handle that properly.\n* UDTF will be asked for the output schema (data types) during the logical plan analyzing.\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #4602 from chenghao-intel/explode_bug and squashes the following commits:\n\nc2a5132 [Cheng Hao] add back resolved for Alias\n556e982 [Cheng Hao] revert the unncessary change\n002c361 [Cheng Hao] change the rule of resolved for Generate\n04ae500 [Cheng Hao] add qualifier only for generator output\n5ee5d2c [Cheng Hao] prepend the new qualifier\nd2e8b43 [Cheng Hao] Update the code as feedback\nca5e7f4 [Cheng Hao] shrink the commits","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/7662ec23bb6c4d4fe4c857b6928eaed0a97d3c04"},{"sha":"f83c0f112d04173f4fc2c5eaf0f9cb11d9180077","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-3386] Share and reuse SerializerInstances in shuffle paths\n\nThis patch modifies several shuffle-related code paths to share and re-use SerializerInstances instead of creating new ones.  Some serializers, such as KryoSerializer or SqlSerializer, can be fairly expensive to create or may consume moderate amounts of memory, so it's probably best to avoid unnecessary serializer creation in hot code paths.\n\nThe key change in this patch is modifying `getDiskWriter()` / `DiskBlockObjectWriter` to accept `SerializerInstance`s instead of `Serializer`s (which are factories for instances).  This allows the disk writer's creator to decide whether the serializer instance can be shared or re-used.\n\nThe rest of the patch modifies several write and read paths to use shared serializers.  One big win is in `ShuffleBlockFetcherIterator`, where we used to create a new serializer per received block.  Similarly, the shuffle write path used to create a new serializer per file even though in many cases only a single thread would be writing to a file at a time.\n\nI made a small serializer reuse optimization in CoarseGrainedExecutorBackend as well, since it seemed like a small and obvious improvement.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #5606 from JoshRosen/SPARK-3386 and squashes the following commits:\n\nf661ce7 [Josh Rosen] Remove thread local; add comment instead\n64f8398 [Josh Rosen] Use ThreadLocal for serializer instance in CoarseGrainedExecutorBackend\naeb680e [Josh Rosen] [SPARK-3386] Reuse SerializerInstance in shuffle code paths","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/f83c0f112d04173f4fc2c5eaf0f9cb11d9180077"},{"sha":"a70e849c7f9e3df5e86113d45b8c4537597cfb29","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[minor] [build] Set java options when generating mima ignores.\n\nThe default java options make the call to GenerateMIMAIgnore take\nforever to run since it's gc'ing all the time. Improve things by\nsetting the perm gen size / max heap size to larger values.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #5615 from vanzin/gen-mima-fix and squashes the following commits:\n\nf44e921 [Marcelo Vanzin] [minor] [build] Set java options when generating mima ignores.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/a70e849c7f9e3df5e86113d45b8c4537597cfb29"},{"sha":"7fe6142cd3c39ec79899878c3deca9d5130d05b1","author":{"email":"8757c4a255384de9fc2175f17a84f948e7081d30@gmail.com","name":"MechCoder"},"message":"[SPARK-6065] [MLlib] Optimize word2vec.findSynonyms using blas calls\n\n1. Use blas calls to find the dot product between two vectors.\n2. Prevent re-computing the L2 norm of the given vector for each word in model.\n\nAuthor: MechCoder <manojkumarsivaraj334@gmail.com>\n\nCloses #5467 from MechCoder/spark-6065 and squashes the following commits:\n\ndd0b0b2 [MechCoder] Preallocate wordVectors\nffc9240 [MechCoder] Minor\n6b74c81 [MechCoder] Switch back to native blas calls\nda1642d [MechCoder] Explicit types and indexing\n64575b0 [MechCoder] Save indexedmap and a wordvecmat instead of matrix\nfbe0108 [MechCoder] Made the following changes 1. Calculate norms during initialization. 2. Use Blas calls from linalg.blas\n1350cf3 [MechCoder] [SPARK-6065] Optimize word2vec.findSynonynms using blas calls","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/7fe6142cd3c39ec79899878c3deca9d5130d05b1"},{"sha":"686dd742e11f6ad0078b7ff9b30b83a118fd8109","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-7036][MLLIB] ALS.train should support DataFrames in PySpark\n\nSchemaRDD works with ALS.train in 1.2, so we should continue support DataFrames for compatibility. coderxiang\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #5619 from mengxr/SPARK-7036 and squashes the following commits:\n\ndfcaf5a [Xiangrui Meng] ALS.train should support DataFrames in PySpark","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/686dd742e11f6ad0078b7ff9b30b83a118fd8109"},{"sha":"ae036d08170202074b266afd17ce34b689c70b0c","author":{"email":"ea5abba9a921ff6c7010543db2857af1b15041e7@usc.edu","name":"Alain"},"message":"[Minor][MLLIB] Fix a minor formatting bug in toString method in Node.scala\n\nadd missing comma and space\n\nAuthor: Alain <aihe@usc.edu>\n\nCloses #5621 from AiHe/tree-node-issue and squashes the following commits:\n\n159a7bb [Alain] [Minor][MLLIB] Fix a minor formatting bug in toString methods in Node.scala\n\n(cherry picked from commit 4508f01890a723f80d631424ff8eda166a13a727)\nSigned-off-by: Xiangrui Meng <meng@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/ae036d08170202074b266afd17ce34b689c70b0c"},{"sha":"b063a61b9852cf9b9d2c905332d2ecb2fd716cc4","author":{"email":"ee744581348450c1d7e87b523bfc73463ff306fd@usu-software.de","name":"mweindel"},"message":"Avoid warning message about invalid refuse_seconds value in Mesos >=0.21...\n\nStarting with version 0.21.0, Apache Mesos is very noisy if the filter parameter refuse_seconds is set to an invalid value like `-1`.\nI have seen systems with millions of log lines like\n```\nW0420 18:00:48.773059 32352 hierarchical_allocator_process.hpp:589] Using the default value of 'refuse_seconds' to create the refused resources filter because the input value is negative\n```\nin the Mesos master INFO and WARNING log files.\nTherefore the CoarseMesosSchedulerBackend should set the default value for refuse seconds (i.e. 5 seconds) directly.\nThis is no problem for the fine-grained MesosSchedulerBackend, as it uses the value 1 second for this parameter.\n\nAuthor: mweindel <m.weindel@usu-software.de>\n\nCloses #5597 from MartinWeindel/master and squashes the following commits:\n\n2f99ffd [mweindel] Avoid warning message about invalid refuse_seconds value in Mesos >=0.21.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/b063a61b9852cf9b9d2c905332d2ecb2fd716cc4"},{"sha":"e72c16e30d85cdc394d318b5551698885cfda9b8","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-6014] [core] Revamp Spark shutdown hooks, fix shutdown races.\n\nThis change adds some new utility code to handle shutdown hooks in\nSpark. The main goal is to take advantage of Hadoop 2.x's API for\nshutdown hooks, which allows Spark to register a hook that will\nrun before the one that cleans up HDFS clients, and thus avoids\nsome races that would cause exceptions to show up and other issues\nsuch as failure to properly close event logs.\n\nUnfortunately, Hadoop 1.x does not have such APIs, so in that case\ncorrectness is still left to chance.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #5560 from vanzin/SPARK-6014 and squashes the following commits:\n\nedfafb1 [Marcelo Vanzin] Better scaladoc.\nfcaeedd [Marcelo Vanzin] Merge branch 'master' into SPARK-6014\ne7039dc [Marcelo Vanzin] [SPARK-6014] [core] Revamp Spark shutdown hooks, fix shutdown races.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/e72c16e30d85cdc394d318b5551698885cfda9b8"}]}