{"push_id":745877971,"size":451,"distinct_size":451,"ref":"refs/heads/master","head":"8be198c86935001907727fd16577231ff776125b","before":"20b474335c68c644150fdc8443a2d0d2dad5e27d","commits":[{"sha":"92540d22e45f9300f413f520a1770e9f36cfa833","author":{"email":"c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","name":"Daoyuan Wang"},"message":"[SPARK-8203] [SPARK-8204] [SQL] conditional function: least/greatest\n\nchenghao-intel zhichao-li qiansl127\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #6851 from adrian-wang/udflg and squashes the following commits:\n\n0f1bff2 [Daoyuan Wang] address comments from davis\n7a6bdbb [Daoyuan Wang] add '.' for hex()\nc1f6824 [Daoyuan Wang] add codegen, test for all types\nec625b0 [Daoyuan Wang] conditional function: least/greatest","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/92540d22e45f9300f413f520a1770e9f36cfa833"},{"sha":"6b89943834a8d9d5d0ecfd97efcc10056d08532a","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-8944][SQL] Support casting between IntervalType and StringType\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #7355 from cloud-fan/fromString and squashes the following commits:\n\n3bbb9d6 [Wenchen Fan] fix code gen\n7dab957 [Wenchen Fan] naming fix\n0fbbe19 [Wenchen Fan] address comments\nac1f3d1 [Wenchen Fan] Support casting between IntervalType and StringType","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/6b89943834a8d9d5d0ecfd97efcc10056d08532a"},{"sha":"a5bc803b7271028e34de1548d55b80ecfb812a7b","author":{"email":"e48675cda3579d10b7ca3cc7e15468574106fa2e@gmail.com","name":"Vincent D. Warmerdam"},"message":"[SPARK-8596] Add module for rstudio link to spark\n\nshivaram, added module for rstudio install\n\nAuthor: Vincent D. Warmerdam <vincentwarmerdam@gmail.com>\n\nCloses #7366 from koaning/rstudio-install and squashes the following commits:\n\ne47c2da [Vincent D. Warmerdam] added rstudio module","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/a5bc803b7271028e34de1548d55b80ecfb812a7b"},{"sha":"7f487c8bde14dbdd244a3493ad11a129ef2bb327","author":{"email":"3f27eab2afd49386f0ac811afb0bc245aa92c0fa@intel.com","name":"Sun Rui"},"message":"[SPARK-6797] [SPARKR] Add support for YARN cluster mode.\n\nThis PR enables SparkR to dynamically ship the SparkR binary package to the AM node in YARN cluster mode, thus it is no longer required that the SparkR package be installed on each worker node.\n\nThis PR uses the JDK jar tool to package the SparkR package, because jar is thought to be available on both Linux/Windows platforms where JDK has been installed.\n\nThis PR does not address the R worker involved in RDD API. Will address it in a separate JIRA issue.\n\nThis PR does not address SBT build. SparkR installation and packaging by SBT will be addressed in a separate JIRA issue.\n\nR/install-dev.bat is not tested. shivaram , Could you help to test it?\n\nAuthor: Sun Rui <rui.sun@intel.com>\n\nCloses #6743 from sun-rui/SPARK-6797 and squashes the following commits:\n\nca63c86 [Sun Rui] Adjust MimaExcludes after rebase.\n7313374 [Sun Rui] Fix unit test errors.\n72695fb [Sun Rui] Fix unit test failures.\n193882f [Sun Rui] Fix Mima test error.\nfe25a33 [Sun Rui] Fix Mima test error.\n35ecfa3 [Sun Rui] Fix comments.\nc38a005 [Sun Rui] Unzipped SparkR binary package is still required for standalone and Mesos modes.\nb05340c [Sun Rui] Fix scala style.\n2ca5048 [Sun Rui] Fix comments.\n1acefd1 [Sun Rui] Fix scala style.\n0aa1e97 [Sun Rui] Fix scala style.\n41d4f17 [Sun Rui] Add support for locating SparkR package for R workers required by RDD APIs.\n49ff948 [Sun Rui] Invoke jar.exe with full path in install-dev.bat.\n7b916c5 [Sun Rui] Use 'rem' consistently.\n3bed438 [Sun Rui] Add a comment.\n681afb0 [Sun Rui] Fix a bug that RRunner does not handle client deployment modes.\ncedfbe2 [Sun Rui] [SPARK-6797][SPARKR] Add support for YARN cluster mode.","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/7f487c8bde14dbdd244a3493ad11a129ef2bb327"},{"sha":"9b62e9375f032548d386aec7468e3d0f7c6da7b2","author":{"email":"8757c4a255384de9fc2175f17a84f948e7081d30@gmail.com","name":"MechCoder"},"message":"[SPARK-8706] [PYSPARK] [PROJECT INFRA] Add pylint checks to PySpark\n\nThis adds Pylint checks to PySpark.\n\nFor now this lazy installs using easy_install to /dev/pylint (similar to the pep8 script).\nWe still need to figure out what rules to be allowed.\n\nAuthor: MechCoder <manojkumarsivaraj334@gmail.com>\n\nCloses #7241 from MechCoder/pylint and squashes the following commits:\n\n8496834 [MechCoder] Silence warnings and make pylint tests fail to check if it works in jenkins\n57393a3 [MechCoder] undefined-variable\na8e2547 [MechCoder] Minor changes\n7753810 [MechCoder] remove trailing whitespace\n75c5d2b [MechCoder] Remove blacklisted arguments and pointless statements check\n6bde250 [MechCoder] Disable all checks for now\n3464666 [MechCoder] Add pylint configuration file\nd28109f [MechCoder] [SPARK-8706] [PySpark] [Project infra] Add pylint checks to PySpark","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/9b62e9375f032548d386aec7468e3d0f7c6da7b2"},{"sha":"5ca26fb64de99fa414dc59ce4cf29a0171894793","author":{"email":"dbe0629cf0118f7dffec38856cad0ec7ac880251@intel.com","name":"Carson Wang"},"message":"[SPARK-8950] [WEBUI] Correct the calculation of SchedulerDelay in StagePage\n\nIn StagePage, the SchedulerDelay is calculated as totalExecutionTime - executorRunTime - executorOverhead - gettingResultTime.\nBut the totalExecutionTime is calculated in the way that doesn't include the gettingResultTime.\n\nAuthor: Carson Wang <carson.wang@intel.com>\n\nCloses #7319 from carsonwang/SchedulerDelayTime and squashes the following commits:\n\nf66fb6e [Carson Wang] Update the code style\n7d971ae [Carson Wang] Correct the calculation of SchedulerDelay","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/5ca26fb64de99fa414dc59ce4cf29a0171894793"},{"sha":"79c35826e626805c5df776b3d22ed37829d239b6","author":{"email":"399a71a219a2bbf15a9ad52c797d50eb921e7e28@gmail.com","name":"Davies Liu"},"message":"Revert \"[SPARK-8706] [PYSPARK] [PROJECT INFRA] Add pylint checks to PySpark\"\n\nThis reverts commit 9b62e9375f032548d386aec7468e3d0f7c6da7b2.","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/79c35826e626805c5df776b3d22ed37829d239b6"},{"sha":"5c41691fe4da0017ccc7e372e3f655dc1c815349","author":{"email":"63b52e08611a2995f34f1279c715920731fa0f6f@users.noreply.github.com","name":"yongtang"},"message":"[SPARK-8954] [BUILD] Remove unneeded deb repository from Dockerfile to fix build error in docker.\n\n[SPARK-8954] [Build]\n1. Remove unneeded deb repository from Dockerfile to fix build error in docker.\n2. Remove unneeded /var/lib/apt/lists/* after install to reduce the docker image size (by ~30MB).\n\nAuthor: yongtang <yongtang@users.noreply.github.com>\n\nCloses #7346 from yongtang/SPARK-8954 and squashes the following commits:\n\n36024a1 [yongtang] [SPARK-8954] [Build] Remove unneeded /var/lib/apt/lists/* after install to reduce the docker image size (by ~30MB)\n7084941 [yongtang] [SPARK-8954] [Build] Remove unneeded deb repository from Dockerfile to fix build error in docker.","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/5c41691fe4da0017ccc7e372e3f655dc1c815349"},{"sha":"714fc55f4aadd5e7b7fb1e462910bfb6a82d9154","author":{"email":"df53206d00b24722989ae08ccb83b71e85ce833d@huawei.com","name":"Vinod K C"},"message":"[SPARK-8991] [ML] Update SharedParamsCodeGen's Generated Documentation\n\nRemoved private[ml] from Generated documentation\n\nAuthor: Vinod K C <vinod.kc@huawei.com>\n\nCloses #7367 from vinodkc/fix_sharedparmascodegen and squashes the following commits:\n\n4fa3c8f [Vinod K C] Adding auto generated code\n7e19025 [Vinod K C] Removed private[ml]","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/714fc55f4aadd5e7b7fb1e462910bfb6a82d9154"},{"sha":"4c797f2b0989317a2d004e5f72a0e593919737ea","author":{"email":"df53206d00b24722989ae08ccb83b71e85ce833d@huawei.com","name":"Vinod K C"},"message":"[SPARK-8636] [SQL] Fix equalNullSafe comparison\n\nAuthor: Vinod K C <vinod.kc@huawei.com>\n\nCloses #7040 from vinodkc/fix_CaseKeyWhen_equalNullSafe and squashes the following commits:\n\nbe5e641 [Vinod K C] Renamed equalNullSafe to threeValueEquals\naac9f67 [Vinod K C] Updated test suite and genCode method\nf2d0b53 [Vinod K C]  Fix equalNullSafe comparison","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/4c797f2b0989317a2d004e5f72a0e593919737ea"},{"sha":"0aed38e4498b24d372bfdc7001959e78536369a1","author":{"email":"e1f8d9790f67825b23faf6a373ab7c83bc8900f2@apache.org","name":"Hari Shreedharan"},"message":"[SPARK-8533] [STREAMING] Upgrade Flume to 1.6.0\n\nAuthor: Hari Shreedharan <hshreedharan@apache.org>\n\nCloses #6939 from harishreedharan/upgrade-flume-1.6.0 and squashes the following commits:\n\n94b80ae [Hari Shreedharan] [SPARK-8533][Streaming] Upgrade Flume to 1.6.0","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/0aed38e4498b24d372bfdc7001959e78536369a1"},{"sha":"b7bcbe25f90ba4e78b548465bc80d4de1d2c4a4a","author":{"email":"8d2d538dbcb9eb2250781c2df783d51543333c0a@cloudera.com","name":"Neelesh Srinivas Salian"},"message":"[SPARK-8743] [STREAMING] Deregister Codahale metrics for streaming when StreamingContext is closed\n\nThe issue link: https://issues.apache.org/jira/browse/SPARK-8743\nDeregister Codahale metrics for streaming when StreamingContext is closed\n\nDesign:\nAdding the method calls in the appropriate start() and stop () methods for the StreamingContext\n\nActions in the PullRequest:\n1) Added the registerSource method call to the start method for the Streaming Context.\n2) Added the removeSource method to the stop method.\n3) Added comments for both 1 and 2 and comment to show initialization of the StreamingSource\n4) Added a test case to check for both registration and de-registration of metrics\n\nPrevious closed PR for reference: https://github.com/apache/spark/pull/7250\n\nAuthor: Neelesh Srinivas Salian <nsalian@cloudera.com>\n\nCloses #7362 from nssalian/branch-SPARK-8743 and squashes the following commits:\n\n7d998a3 [Neelesh Srinivas Salian] Removed the Thread.sleep() call\n8b26397 [Neelesh Srinivas Salian] Moved the scalatest.{} import\n0e8007a [Neelesh Srinivas Salian] moved import org.apache.spark{} to correct place\ndaedaa5 [Neelesh Srinivas Salian] Corrected Ordering of imports\n8873180 [Neelesh Srinivas Salian] Removed redundancy in imports\n59227a4 [Neelesh Srinivas Salian] Changed the ordering of the imports to classify  scala and spark imports\nd8cb577 [Neelesh Srinivas Salian] Added registerSource to start() and removeSource to stop(). Wrote a test to check the registration and de-registration","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/b7bcbe25f90ba4e78b548465bc80d4de1d2c4a4a"},{"sha":"408b384de96b9dbe94945753f7947fbe84272ae1","author":{"email":"9d01406527e5c15536e8dcb4e0c5005dca46786b@netflix.com","name":"Cheolsoo Park"},"message":"[SPARK-6910] [SQL] Support for pushing predicates down to metastore for partition pruning\n\nThis PR supersedes my old one #6921. Since my patch has changed quite a bit, I am opening a new PR to make it easier to review.\n\nThe changes include-\n* Implement `toMetastoreFilter()` function in `HiveShim` that takes `Seq[Expression]` and converts them into a filter string for Hive metastore.\n * This functions matches all the `AttributeReference` + `BinaryComparisonOp` + `Integral/StringType` patterns in `Seq[Expression]` and fold them into a string.\n* Change `hiveQlPartitions` field in `MetastoreRelation` to `getHiveQlPartitions()` function that takes a filter string parameter.\n* Call `getHiveQlPartitions()` in `HiveTableScan` with a filter string.\n\nBut there are some cases in which predicate pushdown is disabled-\n\nCase | Predicate pushdown\n------- | -----------------------------\nHive integral and string types | Yes\nHive varchar type | No\nHive 0.13 and newer | Yes\nHive 0.12 and older | No\nconvertMetastoreParquet=false | Yes\nconvertMetastoreParquet=true | No\n\nIn case of `convertMetastoreParquet=true`, predicates are not pushed down because this conversion happens in an `Analyzer` rule (`HiveMetastoreCatalog.ParquetConversions`). At this point, `HiveTableScan` hasn't run, so predicates are not available. But reading the source code, I think it is intentional to convert the entire Hive table w/ all the partitions into `ParquetRelation` because then `ParquetRelation` can be cached and reused for any query against that table. Please correct me if I am wrong.\n\ncc marmbrus\n\nAuthor: Cheolsoo Park <cheolsoop@netflix.com>\n\nCloses #7216 from piaozhexiu/SPARK-6910-2 and squashes the following commits:\n\naa1490f [Cheolsoo Park] Fix ordering of imports\nc212c4d [Cheolsoo Park] Incorporate review comments\n5e93f9d [Cheolsoo Park] Predicate pushdown into Hive metastore","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/408b384de96b9dbe94945753f7947fbe84272ae1"},{"sha":"20c1434a8dbb25b98f6b434b158ae88e44ce3057","author":{"email":"b91298e12dcfa1e38f5d8e45fc1cad713676d4d6@gmail.com","name":"Joseph Gonzalez"},"message":"[SPARK-9001] Fixing errors in javadocs that lead to failed build/sbt doc\n\nThese are minor corrections in the documentation of several classes that are preventing:\n\n```bash\nbuild/sbt publish-local\n```\n\nI believe this might be an issue associated with running JDK8 as ankurdave does not appear to have this issue in JDK7.\n\nAuthor: Joseph Gonzalez <joseph.e.gonzalez@gmail.com>\n\nCloses #7354 from jegonzal/FixingJavadocErrors and squashes the following commits:\n\n6664b7e [Joseph Gonzalez] making requested changes\n2e16d89 [Joseph Gonzalez] Fixing errors in javadocs that prevents build/sbt publish-local from completing.","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/20c1434a8dbb25b98f6b434b158ae88e44ce3057"},{"sha":"c1feebd8fcba985667db8ccdafd2b5ec76dcfae7","author":{"email":"7aa4d6ed28dd24613aa85968af55a5ebe89c3ab9@haizhi.com","name":"zhaishidan"},"message":"[SPARK-9010] [DOCUMENTATION] Improve the Spark Configuration document about `spark.kryoserializer.buffer`\n\nThe meaning of spark.kryoserializer.buffer should be \"Initial size of Kryo's serialization buffer. Note that there will be one buffer per core on each worker. This buffer will grow up to spark.kryoserializer.buffer.max if needed.\".\n\nThe spark.kryoserializer.buffer.max.mb is out-of-date in spark 1.4.\n\nAuthor: zhaishidan <zhaishidan@haizhi.com>\n\nCloses #7393 from stanzhai/master and squashes the following commits:\n\n69729ef [zhaishidan] fix document error about spark.kryoserializer.buffer.max.mb","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/c1feebd8fcba985667db8ccdafd2b5ec76dcfae7"},{"sha":"257236c3e17906098f801cbc2059e7a9054e8cab","author":{"email":"c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","name":"Daoyuan Wang"},"message":"[SPARK-6851] [SQL] function least/greatest follow up\n\nThis is a follow up of remaining comments from #6851\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #7387 from adrian-wang/udflgfollow and squashes the following commits:\n\n6163e62 [Daoyuan Wang] add skipping null values\ne8c2e09 [Daoyuan Wang] use seq\n8362966 [Daoyuan Wang] pr6851 follow up","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/257236c3e17906098f801cbc2059e7a9054e8cab"},{"sha":"59d820aa8dec08b744971237860b4c6bef577ddf","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-9029] [SQL] shortcut CaseKeyWhen if key is null\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #7389 from cloud-fan/case-when and squashes the following commits:\n\nea4b6ba [Wenchen Fan] shortcut for case key when","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/59d820aa8dec08b744971237860b4c6bef577ddf"},{"sha":"37f2d9635ff874fb8ad9d246e49faf6098d501c3","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-9027] [SQL] Generalize metastore predicate pushdown\n\nAdd support for pushing down metastore filters that are in different orders and add some unit tests.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #7386 from marmbrus/metastoreFilters and squashes the following commits:\n\n05a4524 [Michael Armbrust] [SPARK-9027][SQL] Generalize metastore predicate pushdown","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/37f2d9635ff874fb8ad9d246e49faf6098d501c3"},{"sha":"c4e98ff066cc6f0839d15140eb471d74a0d83e91","author":{"email":"eaca3ca4b51928dd768a043ece15380fadcb1b95@capitalone.com","name":"Brennon York"},"message":"[SPARK-8933] [BUILD] Provide a --force flag to build/mvn that always uses downloaded maven\n\nadded --force flag to manually download, if necessary, and use a built-in version of maven best for spark\n\nAuthor: Brennon York <brennon.york@capitalone.com>\n\nCloses #7374 from brennonyork/SPARK-8933 and squashes the following commits:\n\nd673127 [Brennon York] added --force flag to manually download, if necessary, and use a built-in version of maven best for spark","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/c4e98ff066cc6f0839d15140eb471d74a0d83e91"},{"sha":"8fb3a65cbb714120d612e58ef9d12b0521a83260","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-8911] Fix local mode endless heartbeats\n\nAs of #7173 we expect executors to properly register with the driver before responding to their heartbeats. This behavior is not matched in local mode. This patch adds the missing event that needs to be posted.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #7382 from andrewor14/fix-local-heartbeat and squashes the following commits:\n\n1258bdf [Andrew Or] Post ExecutorAdded event to local executor","distinct":true,"url":"https://api.github.com/repos/koaning/spark/commits/8fb3a65cbb714120d612e58ef9d12b0521a83260"}]}