{"push_id":874442269,"size":78,"distinct_size":78,"ref":"refs/heads/master","head":"a2dce22e0a25922e2052318d32f32877b7c27ec2","before":"1429e0a2b562469146b6fa06051c85a00092e5b8","commits":[{"sha":"3a6807fdf07b0e73d76502a6bd91ad979fde8b61","author":{"email":"ced9a24e2e494bb60341a9f26a61a78e7a11abad@apache.org","name":"Jeff Zhang"},"message":"[SPARK-11804] [PYSPARK] Exception raise when using Jdbc predicates opt…\n\n…ion in PySpark\n\nAuthor: Jeff Zhang <zjffdu@apache.org>\n\nCloses #9791 from zjffdu/SPARK-11804.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/3a6807fdf07b0e73d76502a6bd91ad979fde8b61"},{"sha":"a97d6f3a5861e9f2bbe36957e3b39f835f3e214c","author":{"email":"ffad4dc97bb49db436733c212788f8b59c50dd2b@gmail.com","name":"zero323"},"message":"[SPARK-11281][SPARKR] Add tests covering the issue.\n\nThe goal of this PR is to add tests covering the issue to ensure that is was resolved by [SPARK-11086](https://issues.apache.org/jira/browse/SPARK-11086).\n\nAuthor: zero323 <matthew.szymkiewicz@gmail.com>\n\nCloses #9743 from zero323/SPARK-11281-tests.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/a97d6f3a5861e9f2bbe36957e3b39f835f3e214c"},{"sha":"224723e6a8b198ef45d6c5ca5d2f9c61188ada8f","author":{"email":"3f27eab2afd49386f0ac811afb0bc245aa92c0fa@intel.com","name":"Sun Rui"},"message":"[SPARK-11773][SPARKR] Implement collection functions in SparkR.\n\nAuthor: Sun Rui <rui.sun@intel.com>\n\nCloses #9764 from sun-rui/SPARK-11773.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/224723e6a8b198ef45d6c5ca5d2f9c61188ada8f"},{"sha":"3cca5ffb3d60d5de9a54bc71cf0b8279898936d2","author":{"email":"de2d98d5e51044388d2f86db02e49f9c05066048@gmail.com","name":"Hurshal Patel"},"message":"[SPARK-11195][CORE] Use correct classloader for TaskResultGetter\n\nMake sure we are using the context classloader when deserializing failed TaskResults instead of the Spark classloader.\n\nThe issue is that `enqueueFailedTask` was using the incorrect classloader which results in `ClassNotFoundException`.\n\nAdds a test in TaskResultGetterSuite that compiles a custom exception, throws it on the executor, and asserts that Spark handles the TaskResult deserialization instead of returning `UnknownReason`.\n\nSee #9367 for previous comments\nSee SPARK-11195 for a full repro\n\nAuthor: Hurshal Patel <hpatel516@gmail.com>\n\nCloses #9779 from choochootrain/spark-11195-master.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/3cca5ffb3d60d5de9a54bc71cf0b8279898936d2"},{"sha":"cffb899c4397ecccedbcc41e7cf3da91f953435a","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11803][SQL] fix Dataset self-join\n\nWhen we resolve the join operator, we may change the output of right side if self-join is detected. So in `Dataset.joinWith`, we should resolve the join operator first, and then get the left output and right output from it, instead of using `left.output` and `right.output` directly.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9806 from cloud-fan/self-join.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/cffb899c4397ecccedbcc41e7cf3da91f953435a"},{"sha":"33b837333435ceb0c04d1f361a5383c4fe6a5a75","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11725][SQL] correctly handle null inputs for UDF\n\nIf user use primitive parameters in UDF, there is no way for him to do the null-check for primitive inputs, so we are assuming the primitive input is null-propagatable for this case and return null if the input is null.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9770 from cloud-fan/udf.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/33b837333435ceb0c04d1f361a5383c4fe6a5a75"},{"sha":"dbf428c87ab34b6f76c75946043bdf5f60c9b1b3","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11795][SQL] combine grouping attributes into a single NamedExpression\n\nwe use `ExpressionEncoder.tuple` to build the result encoder, which assumes the input encoder should point to a struct type field if it’s non-flat.\nHowever, our keyEncoder always point to a flat field/fields: `groupingAttributes`, we should combine them into a single `NamedExpression`.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9792 from cloud-fan/agg.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/dbf428c87ab34b6f76c75946043bdf5f60c9b1b3"},{"sha":"90a7519daaa7f4ee3be7c5a9aa244120811ff6eb","author":{"email":"249f1848d407dd74af9592a86811fcca179f01b2@gmail.com","name":"Jakob Odersky"},"message":"[MINOR][BUILD] Ignore ensime cache\n\nUsing ENSIME, I often have `.ensime_cache` polluting my source tree. This PR simply adds the cache directory to `.gitignore`\n\nAuthor: Jakob Odersky <jodersky@gmail.com>\n\nCloses #9708 from jodersky/master.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/90a7519daaa7f4ee3be7c5a9aa244120811ff6eb"},{"sha":"6f99522d13d8db9fcc767f7c3189557b9a53d283","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-11792] [SQL] [FOLLOW-UP] Change SizeEstimation to KnownSizeEstimation and make estimatedSize return Long instead of Option[Long]\n\nhttps://issues.apache.org/jira/browse/SPARK-11792\n\nThe main changes include:\n* Renaming `SizeEstimation` to `KnownSizeEstimation`. Hopefully this new name has more information.\n* Making `estimatedSize` return `Long` instead of `Option[Long]`.\n* In `UnsaveHashedRelation`, `estimatedSize` will delegate the work to `SizeEstimator` if we have not created a `BytesToBytesMap`.\n\nSince we will put `UnsaveHashedRelation` to `BlockManager`, it is generally good to let it provide a more accurate size estimation. Also, if we do not put `BytesToBytesMap` directly into `BlockerManager`, I feel it is not really necessary to make `BytesToBytesMap` extends `KnownSizeEstimation`.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #9813 from yhuai/SPARK-11792-followup.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/6f99522d13d8db9fcc767f7c3189557b9a53d283"},{"sha":"94624eacb0fdbbe210894151a956f8150cdf527e","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-11739][SQL] clear the instantiated SQLContext\n\nCurrently, if the first SQLContext is not removed after stopping SparkContext, a SQLContext could set there forever. This patch make this more robust.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #9706 from davies/clear_context.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/94624eacb0fdbbe210894151a956f8150cdf527e"},{"sha":"31921e0f0bd559d042148d1ea32f865fb3068f38","author":{"email":"0510f24c259a4f603c85979def2fe3bfbef1c219@us.ibm.com","name":"Bryan Cutler"},"message":"[SPARK-4557][STREAMING] Spark Streaming foreachRDD Java API method should accept a VoidFunction<...>\n\nCurrently streaming foreachRDD Java API uses a function prototype requiring a return value of null.  This PR deprecates the old method and uses VoidFunction to allow for more concise declaration.  Also added VoidFunction2 to Java API in order to use in Streaming methods.  Unit test is added for using foreachRDD with VoidFunction, and changes have been tested with Java 7 and Java 8 using lambdas.\n\nAuthor: Bryan Cutler <bjcutler@us.ibm.com>\n\nCloses #9488 from BryanCutler/foreachRDD-VoidFunction-SPARK-4557.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/31921e0f0bd559d042148d1ea32f865fb3068f38"},{"sha":"a416e41e285700f861559d710dbf857405bfddf6","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11809] Switch the default Mesos mode to coarse-grained mode\n\nBased on my conversions with people, I believe the consensus is that the coarse-grained mode is more stable and easier to reason about. It is best to use that as the default rather than the more flaky fine-grained mode.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9795 from rxin/SPARK-11809.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/a416e41e285700f861559d710dbf857405bfddf6"},{"sha":"7c5b641808740ba5eed05ba8204cdbaf3fc579f5","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-10745][CORE] Separate configs between shuffle and RPC\n\n[SPARK-6028](https://issues.apache.org/jira/browse/SPARK-6028) uses network module to implement RPC. However, there are some configurations named with `spark.shuffle` prefix in the network module.\n\nThis PR refactors them to make sure the user can control them in shuffle and RPC separately. The user can use `spark.rpc.*` to set the configuration for netty RPC.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #9481 from zsxwing/SPARK-10745.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/7c5b641808740ba5eed05ba8204cdbaf3fc579f5"},{"sha":"09ad9533d5760652de59fa4830c24cb8667958ac","author":{"email":"081ca5f8e359b969a806e4ba1eafbafb637fc7cb@gmail.com","name":"JihongMa"},"message":"[SPARK-11720][SQL][ML] Handle edge cases when count = 0 or 1 for Stats function\n\nreturn Double.NaN for mean/average when count == 0 for all numeric types that is converted to Double, Decimal type continue to return null.\n\nAuthor: JihongMa <linlin200605@gmail.com>\n\nCloses #9705 from JihongMA/SPARK-11720.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/09ad9533d5760652de59fa4830c24cb8667958ac"},{"sha":"045a4f045821dcf60442f0600c2df1b79bddb536","author":{"email":"6281cb803d7b4c4670f4af50faec23b8dd812d9f@163.com","name":"Wenjian Huang"},"message":"[SPARK-6790][ML] Add spark.ml LinearRegression import/export\n\nThis replaces [https://github.com/apache/spark/pull/9656] with updates.\n\nfayeshine should be the main author when this PR is committed.\n\nCC: mengxr fayeshine\n\nAuthor: Wenjian Huang <nextrush@163.com>\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #9814 from jkbradley/fayeshine-patch-6790.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/045a4f045821dcf60442f0600c2df1b79bddb536"},{"sha":"2acdf10b1f3bb1242dba64efa798c672fde9f0d2","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-6789][ML] Add Readable, Writable support for spark.ml ALS, ALSModel\n\nAlso modifies DefaultParamsWriter.saveMetadata to take optional extra metadata.\n\nCC: mengxr yanboliang\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #9786 from jkbradley/als-io.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/2acdf10b1f3bb1242dba64efa798c672fde9f0d2"},{"sha":"e391abdf2cb6098a35347bd123b815ee9ac5b689","author":{"email":"66332c93c294b372ab1bdd5acb9b89a538e2d2f2@gmail.com","name":"Yuhao Yang"},"message":"[SPARK-11813][MLLIB] Avoid serialization of vocab in Word2Vec\n\njira: https://issues.apache.org/jira/browse/SPARK-11813\n\nI found the problem during training a large corpus. Avoid serialization of vocab in Word2Vec has 2 benefits.\n1. Performance improvement for less serialization.\n2. Increase the capacity of Word2Vec a lot.\nCurrently in the fit of word2vec, the closure mainly includes serialization of Word2Vec and 2 global table.\nthe main part of Word2vec is the vocab of size: vocab * 40 * 2 * 4 = 320 vocab\n2 global table: vocab * vectorSize * 8. If vectorSize = 20, that's 160 vocab.\n\nTheir sum cannot exceed Int.max due to the restriction of ByteArrayOutputStream. In any case, avoiding serialization of vocab helps decrease the size of the closure serialization, especially when vectorSize is small, thus to allow larger vocabulary.\n\nActually there's another possible fix, make local copy of fields to avoid including Word2Vec in the closure. Let me know if that's preferred.\n\nAuthor: Yuhao Yang <hhbyyh@gmail.com>\n\nCloses #9803 from hhbyyh/w2vVocab.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/e391abdf2cb6098a35347bd123b815ee9ac5b689"},{"sha":"e222d758499ad2609046cc1a2cc8afb45c5bccbb","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-11684][R][ML][DOC] Update SparkR glm API doc, user guide and example codes\n\nThis PR includes:\n* Update SparkR:::glm, SparkR:::summary API docs.\n* Update SparkR machine learning user guide and example codes to show:\n  * supporting feature interaction in R formula.\n  * summary for gaussian GLM model.\n  * coefficients for binomial GLM model.\n\nmengxr\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #9727 from yanboliang/spark-11684.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/e222d758499ad2609046cc1a2cc8afb45c5bccbb"},{"sha":"603a721c21488e17c15c45ce1de893e6b3d02274","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-11820][ML][PYSPARK] PySpark LiR & LoR should support weightCol\n\n[SPARK-7685](https://issues.apache.org/jira/browse/SPARK-7685) and [SPARK-9642](https://issues.apache.org/jira/browse/SPARK-9642) have already supported setting weight column for ```LogisticRegression``` and ```LinearRegression```. It's a very important feature, PySpark should also support. mengxr\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #9811 from yanboliang/spark-11820.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/603a721c21488e17c15c45ce1de893e6b3d02274"},{"sha":"54db79702513e11335c33bcf3a03c59e965e6f16","author":{"email":"ee7669731148a67ed053406d4e2088f2bc2c2de0@us.ibm.com","name":"Dilip Biswal"},"message":"[SPARK-11544][SQL] sqlContext doesn't use PathFilter\n\nApply the user supplied pathfilter while retrieving the files from fs.\n\nAuthor: Dilip Biswal <dbiswal@us.ibm.com>\n\nCloses #9652 from dilipbiswal/spark-11544.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/54db79702513e11335c33bcf3a03c59e965e6f16"}]}