{"push_id":848809685,"size":62,"distinct_size":62,"ref":"refs/heads/master","head":"f8d93edec82eedab59d50aec06ca2de7e4cf14f6","before":"8b292b19c9b3aaaa51b919a12132e099e5be832d","commits":[{"sha":"d77d198fcc7c532a699f062a3e3877a7679809da","author":{"email":"7853928d03425cf3be35be2fa0c5b99641435d26@gmail.com","name":"Xusen Yin"},"message":"[SPARK-11297] Add new code tags\n\nmengxr https://issues.apache.org/jira/browse/SPARK-11297\n\nAdd new code tags to hold the same look and feel with previous documents.\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\n\nCloses #9265 from yinxusen/SPARK-11297.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/d77d198fcc7c532a699f062a3e3877a7679809da"},{"sha":"feb8d6a44fbfc31a880aaaac0cfcaadc91786073","author":{"email":"915bc03a8ff2a360f7a15821a27f5c2b2e1d5d65@site2mobile.com","name":"Sem Mulder"},"message":"[SPARK-11276][CORE] SizeEstimator prevents class unloading\n\nThe SizeEstimator keeps a cache of ClassInfos but this cache uses Class objects as keys.\nWhich results in strong references to the Class objects. If these classes are dynamically created\nthis prevents the corresponding ClassLoader from being GCed. Leading to PermGen exhaustion.\n\nWe use a Map with WeakKeys to prevent this issue.\n\nAuthor: Sem Mulder <sem.mulder@site2mobile.com>\n\nCloses #9244 from SemMulder/fix-sizeestimator-classunloading.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/feb8d6a44fbfc31a880aaaac0cfcaadc91786073"},{"sha":"8f888eea1aef5a28916ec406a99fc19648681ecf","author":{"email":"b1c1d8736f20db3fb6c1c66bb1455ed43909f0d8@nicolasevans.org","name":"Nick Evans"},"message":"[SPARK-11270][STREAMING] Add improved equality testing for TopicAndPartition from the Kafka Streaming API\n\njerryshao tdas\n\nI know this is kind of minor, and I know you all are busy, but this brings this class in line with the `OffsetRange` class, and makes tests a little more concise.\n\nInstead of doing something like:\n```\nassert topic_and_partition_instance._topic == \"foo\"\nassert topic_and_partition_instance._partition == 0\n```\n\nYou can do something like:\n```\nassert topic_and_partition_instance == TopicAndPartition(\"foo\", 0)\n```\n\nBefore:\n```\n>>> from pyspark.streaming.kafka import TopicAndPartition\n>>> TopicAndPartition(\"foo\", 0) == TopicAndPartition(\"foo\", 0)\nFalse\n```\n\nAfter:\n```\n>>> from pyspark.streaming.kafka import TopicAndPartition\n>>> TopicAndPartition(\"foo\", 0) == TopicAndPartition(\"foo\", 0)\nTrue\n```\n\nI couldn't find any tests - am I missing something?\n\nAuthor: Nick Evans <me@nicolasevans.org>\n\nCloses #9236 from manygrams/topic_and_partition_equality.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/8f888eea1aef5a28916ec406a99fc19648681ecf"},{"sha":"17f499920776e0e995434cfa300ff2ff38658fa8","author":{"email":"869b3d745579883f35f52fd5c638c218643dcad6@gmail.com","name":"maxwell"},"message":"[SPARK-5569][STREAMING] fix ObjectInputStreamWithLoader for supporting load array classes.\n\nWhen use Kafka DirectStream API to create checkpoint and restore saved checkpoint when restart,\nClassNotFound exception would occur.\n\nThe reason for this error is that ObjectInputStreamWithLoader extends the ObjectInputStream class and override its resolveClass method. But Instead of Using Class.forName(desc,false,loader), Spark uses loader.loadClass(desc) to instance the class, which do not works with array class.\n\nFor example:\nClass.forName(\"[Lorg.apache.spark.streaming.kafka.OffsetRange.\",false,loader) works well while loader.loadClass(\"[Lorg.apache.spark.streaming.kafka.OffsetRange\") would throw an class not found exception.\n\ndetails of the difference between Class.forName and loader.loadClass can be found here.\nhttp://bugs.java.com/view_bug.do?bug_id=6446627\n\nAuthor: maxwell <maxwellzdm@gmail.com>\nAuthor: DEMING ZHU <deming.zhu@linecorp.com>\n\nCloses #8955 from maxwellzdm/master.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/17f499920776e0e995434cfa300ff2ff38658fa8"},{"sha":"958a0ec8fa58ff091f595db2b574a7aa3ff41253","author":{"email":"7f11b58c6a4da8c2d1d553e71b11838fbc473d8a@us.ibm.com","name":"Jia Li"},"message":"[SPARK-11277][SQL] sort_array throws exception scala.MatchError\n\nI'm new to spark. I was trying out the sort_array function then hit this exception. I looked into the spark source code. I found the root cause is that sort_array does not check for an array of NULLs. It's not meaningful to sort an array of entirely NULLs anyway.\n\nI'm adding a check on the input array type to SortArray. If the array consists of NULLs entirely, there is no need to sort such array. I have also added a test case for this.\n\nPlease help to review my fix. Thanks!\n\nAuthor: Jia Li <jiali@us.ibm.com>\n\nCloses #9247 from jliwork/SPARK-11277.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/958a0ec8fa58ff091f595db2b574a7aa3ff41253"},{"sha":"360ed832f5213b805ac28cf1d2828be09480f2d6","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-11303][SQL] filter should not be pushed down into sample\n\nWhen sampling and then filtering DataFrame, the SQL Optimizer will push down filter into sample and produce wrong result. This is due to the sampler is calculated based on the original scope rather than the scope after filtering.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #9294 from yanboliang/spark-11303.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/360ed832f5213b805ac28cf1d2828be09480f2d6"},{"sha":"9fc16a82adb5f3db2a250765c11393794404a51b","author":{"email":"284a00ea9901227c42561196ff8576aed3ac5631@gmail.com","name":"Kay Ousterhout"},"message":"[SPARK-11306] Fix hang when JVM exits.\n\nThis commit fixes a bug where, in Standalone mode, if a task fails and crashes the JVM, the\nfailure is considered a \"normal failure\" (meaning it's considered unrelated to the task), so\nthe failure isn't counted against the task's maximum number of failures:\nhttps://github.com/apache/spark/commit/af3bc59d1f5d9d952c2d7ad1af599c49f1dbdaf0#diff-a755f3d892ff2506a7aa7db52022d77cL138.\nAs a result, if a task fails in a way that results in it crashing the JVM, it will continuously be\nre-launched, resulting in a hang. This commit fixes that problem.\n\nThis bug was introduced by #8007; andrewor14 mccheah vanzin can you take a look at this?\n\nThis error is hard to trigger because we handle executor losses through 2 code paths (the second is via Akka, where Akka notices that the executor endpoint is disconnected).  In my setup, the Akka code path completes first, and doesn't have this bug, so things work fine (see my recent email to the dev list about this).  If I manually disable the Akka code path, I can see the hang (and this commit fixes the issue).\n\nAuthor: Kay Ousterhout <kayousterhout@gmail.com>\n\nCloses #9273 from kayousterhout/SPARK-11306.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/9fc16a82adb5f3db2a250765c11393794404a51b"},{"sha":"3bdbbc6c972567861044dd6a6dc82f35cd12442d","author":{"email":"3c26cc6a278bab57a956b43cbc3c2947e24baa4a@us.ibm.com","name":"Mike Dusenberry"},"message":"[SPARK-6488][MLLIB][PYTHON] Support addition/multiplication in PySpark's BlockMatrix\n\nThis PR adds addition and multiplication to PySpark's `BlockMatrix` class via `add` and `multiply` functions.\n\nAuthor: Mike Dusenberry <mwdusenb@us.ibm.com>\n\nCloses #9139 from dusenberrymw/SPARK-6488_Add_Addition_and_Multiplication_to_PySpark_BlockMatrix.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/3bdbbc6c972567861044dd6a6dc82f35cd12442d"},{"sha":"5a5f65905a202e59bc85170b01c57a883718ddf6","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-11347] [SQL] Support for joinWith in Datasets\n\nThis PR adds a new operation `joinWith` to a `Dataset`, which returns a `Tuple` for each pair where a given `condition` evaluates to true.\n\n```scala\ncase class ClassData(a: String, b: Int)\n\nval ds1 = Seq(ClassData(\"a\", 1), ClassData(\"b\", 2)).toDS()\nval ds2 = Seq((\"a\", 1), (\"b\", 2)).toDS()\n\n> ds1.joinWith(ds2, $\"_1\" === $\"a\").collect()\nres0: Array((ClassData(\"a\", 1), (\"a\", 1)), (ClassData(\"b\", 2), (\"b\", 2)))\n```\n\nThis operation is similar to the relation `join` function with one important difference in the result schema. Since `joinWith` preserves objects present on either side of the join, the result schema is similarly nested into a tuple under the column names `_1` and `_2`.\n\nThis type of join can be useful both for preserving type-safety with the original object types as well as working with relational data where either side of the join has column names in common.\n\n## Required Changes to Encoders\nIn the process of working on this patch, several deficiencies to the way that we were handling encoders were discovered.  Specifically, it turned out to be very difficult to `rebind` the non-expression based encoders to extract the nested objects from the results of joins (and also typed selects that return tuples).\n\nAs a result the following changes were made.\n - `ClassEncoder` has been renamed to `ExpressionEncoder` and has been improved to also handle primitive types.  Additionally, it is now possible to take arbitrary expression encoders and rewrite them into a single encoder that returns a tuple.\n - All internal operations on `Dataset`s now require an `ExpressionEncoder`.  If the users tries to pass a non-`ExpressionEncoder` in, an error will be thrown.  We can relax this requirement in the future by constructing a wrapper class that uses expressions to project the row to the expected schema, shielding the users code from the required remapping.  This will give us a nice balance where we don't force user encoders to understand attribute references and binding, but still allow our native encoder to leverage runtime code generation to construct specific encoders for a given schema that avoid an extra remapping step.\n - Additionally, the semantics for different types of objects are now better defined.  As stated in the `ExpressionEncoder` scaladoc:\n  - Classes will have their sub fields extracted by name using `UnresolvedAttribute` expressions\n  and `UnresolvedExtractValue` expressions.\n  - Tuples will have their subfields extracted by position using `BoundReference` expressions.\n  - Primitives will have their values extracted from the first ordinal with a schema that defaults\n  to the name `value`.\n - Finally, the binding lifecycle for `Encoders` has now been unified across the codebase.  Encoders are now `resolved` to the appropriate schema in the constructor of `Dataset`.  This process replaces an unresolved expressions with concrete `AttributeReference` expressions.  Binding then happens on demand, when an encoder is going to be used to construct an object.  This closely mirrors the lifecycle for standard expressions when executing normal SQL or `DataFrame` queries.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #9300 from marmbrus/datasets-tuples.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/5a5f65905a202e59bc85170b01c57a883718ddf6"},{"sha":"9dba5fb2b59174cefde5b62a5c892fe5925bea38","author":{"email":"a8225d365811cb0449b395490c087d0ddd3c6c6f@gmail.com","name":"vectorijk"},"message":"[SPARK-10024][PYSPARK] Python API RF and GBT related params clear up\n\nimplement {RandomForest, GBT, TreeEnsemble, TreeClassifier, TreeRegressor}Params for Python API\nin pyspark/ml/{classification, regression}.py\n\nAuthor: vectorijk <jiangkai@gmail.com>\n\nCloses #9233 from vectorijk/spark-10024.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/9dba5fb2b59174cefde5b62a5c892fe5925bea38"},{"sha":"4f030b9e82172659d250281782ac573cbd1438fc","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":"[SPARK-11324][STREAMING] Flag for closing Write Ahead Logs after a write\n\nCurrently the Write Ahead Log in Spark Streaming flushes data as writes need to be made. S3 does not support flushing of data, data is written once the stream is actually closed.\nIn case of failure, the data for the last minute (default rolling interval) will not be properly written. Therefore we need a flag to close the stream after the write, so that we achieve read after write consistency.\n\ncc tdas zsxwing\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #9285 from brkyvz/caw-wal.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/4f030b9e82172659d250281782ac573cbd1438fc"},{"sha":"9fbd75ab5d46612e52116ec5b9ced70715cf26b5","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-11212][CORE][STREAMING] Make preferred locations support ExecutorCacheTaskLocation and update…\n\n… ReceiverTracker and ReceiverSchedulingPolicy to use it\n\nThis PR includes the following changes:\n\n1. Add a new preferred location format, `executor_<host>_<executorID>` (e.g., \"executor_localhost_2\"), to support specifying the executor locations for RDD.\n2. Use the new preferred location format in `ReceiverTracker` to optimize the starting time of Receivers when there are multiple executors in a host.\n\nThe goal of this PR is to enable the streaming scheduler to place receivers (which run as tasks) in specific executors. Basically, I want to have more control on the placement of the receivers such that they are evenly distributed among the executors. We tried to do this without changing the core scheduling logic. But it does not allow specifying particular executor as preferred location, only at the host level. So if there are two executors in the same host, and I want two receivers to run on them (one on each executor), I cannot specify that. Current code only specifies the host as preference, which may end up launching both receivers on the same executor. We try to work around it but restarting a receiver when it does not launch in the desired executor and hope that next time it will be started in the right one. But that cause lots of restarts, and delays in correctly launching the receiver.\n\nSo this change, would allow the streaming scheduler to specify the exact executor as the preferred location. Also this is not exposed to the user, only the streaming scheduler uses this.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #9181 from zsxwing/executor-location.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/9fbd75ab5d46612e52116ec5b9ced70715cf26b5"},{"sha":"b960a890561eaf3795b93c621bd95be81e56f5b7","author":{"email":"284a00ea9901227c42561196ff8576aed3ac5631@gmail.com","name":"Kay Ousterhout"},"message":"[SPARK-11178] Improving naming around task failures.\n\nCommit af3bc59d1f5d9d952c2d7ad1af599c49f1dbdaf0 introduced new\nfunctionality so that if an executor dies for a reason that's not\ncaused by one of the tasks running on the executor (e.g., due to\npre-emption), Spark doesn't count the failure towards the maximum\nnumber of failures for the task.  That commit introduced some vague\nnaming that this commit attempts to fix; in particular:\n\n(1) The variable \"isNormalExit\", which was used to refer to cases where\nthe executor died for a reason unrelated to the tasks running on the\nmachine, has been renamed (and reversed) to \"exitCausedByApp\". The problem\nwith the existing name is that it's not clear (at least to me!) what it\nmeans for an exit to be \"normal\"; the new name is intended to make the\npurpose of this variable more clear.\n\n(2) The variable \"shouldEventuallyFailJob\" has been renamed to\n\"countTowardsTaskFailures\". This variable is used to determine whether\na task's failure should be counted towards the maximum number of failures\nallowed for a task before the associated Stage is aborted. The problem\nwith the existing name is that it can be confused with implying that\nthe task's failure should immediately cause the stage to fail because it\nis somehow fatal (this is the case for a fetch failure, for example: if\na task fails because of a fetch failure, there's no point in retrying,\nand the whole stage should be failed).\n\nAuthor: Kay Ousterhout <kayousterhout@gmail.com>\n\nCloses #9164 from kayousterhout/SPARK-11178.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/b960a890561eaf3795b93c621bd95be81e56f5b7"},{"sha":"d9c6039897236c3f1e4503aa95c5c9b07b32eadd","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-10484] [SQL] Optimize the cartesian join with broadcast join for some cases\n\nIn some cases, we can broadcast the smaller relation in cartesian join, which improve the performance significantly.\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #8652 from chenghao-intel/cartesian.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/d9c6039897236c3f1e4503aa95c5c9b07b32eadd"},{"sha":"826e1e304b57abbc56b8b7ffd663d53942ab3c7c","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-11302][MLLIB] 2) Multivariate Gaussian Model with Covariance matrix returns incorrect answer in some cases\n\nFix computation of root-sigma-inverse in multivariate Gaussian; add a test and fix related Python mixture model test.\n\nSupersedes https://github.com/apache/spark/pull/9293\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #9309 from srowen/SPARK-11302.2.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/826e1e304b57abbc56b8b7ffd663d53942ab3c7c"},{"sha":"82c1c5772817785709b0289f7d836beba812c791","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[MINOR][ML] fix compile warns\n\nThis fixes some compile time warnings.\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #9319 from mengxr/mllib-compile-warn-20151027.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/82c1c5772817785709b0289f7d836beba812c791"},{"sha":"5f1cee6f158adb1f9f485ed1d529c56bace68adc","author":{"email":"0b0deb5eb3f500d85a72e7cd9257ea267c74c59b@us.ibm.com","name":"Nakul Jindal"},"message":"[SPARK-11332] [ML] Refactored to use ml.feature.Instance instead of WeightedLeastSquare.Instance\n\nWeightedLeastSquares now uses the common Instance class in ml.feature instead of a private one.\n\nAuthor: Nakul Jindal <njindal@us.ibm.com>\n\nCloses #9325 from nakul02/SPARK-11332_refactor_WeightedLeastSquares_dot_Instance.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/5f1cee6f158adb1f9f485ed1d529c56bace68adc"},{"sha":"075ce4914fdcbbcc7286c3c30cb940ed28d474d2","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11313][SQL] implement cogroup on DataSets (support 2 datasets)\n\nA simpler version of https://github.com/apache/spark/pull/9279, only support 2 datasets.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9324 from cloud-fan/cogroup2.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/075ce4914fdcbbcc7286c3c30cb940ed28d474d2"},{"sha":"fd9e345ceeff385ba614a16d478097650caa98d0","author":{"email":"b224c921725891305d717e8367b0474badb9f3be@gmail.com","name":"Mageswaran.D"},"message":"Typo in mllib-evaluation-metrics.md\n\nRecall by threshold snippet was using \"precisionByThreshold\"\n\nAuthor: Mageswaran.D <mageswaran1989@gmail.com>\n\nCloses #9333 from Mageswaran1989/Typo_in_mllib-evaluation-metrics.md.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/fd9e345ceeff385ba614a16d478097650caa98d0"},{"sha":"fba9e95452ca0a9b589bc14b27c750c69f482b8d","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-11369][ML][R] SparkR glm should support setting standardize\n\nSparkR glm currently support :\n```formula, family = c(“gaussian”, “binomial”), data, lambda = 0, alpha = 0```\nWe should also support setting standardize which has been defined at [design documentation](https://docs.google.com/document/d/10NZNSEurN2EdWM31uFYsgayIPfCFHiuIu3pCWrUmP_c/edit)\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #9331 from yanboliang/spark-11369.","distinct":true,"url":"https://api.github.com/repos/pramodl/spark/commits/fba9e95452ca0a9b589bc14b27c750c69f482b8d"}]}