{"push_id":800510799,"size":10,"distinct_size":10,"ref":"refs/heads/master","head":"0bd0e5bed2176b119b3ada590993e153757ea09b","before":"c986e933a900602af47966bd41edb2116c421a39","commits":[{"sha":"1cd67415728e660a90e4dbe136272b5d6b8f1142","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-9821] [PYSPARK] pyspark-reduceByKey-should-take-a-custom-partitioner\n\nfrom the issue:\n\nIn Scala, I can supply a custom partitioner to reduceByKey (and other aggregation/repartitioning methods like aggregateByKey and combinedByKey), but as far as I can tell from the Pyspark API, there's no way to do the same in Python.\nHere's an example of my code in Scala:\nweblogs.map(s => (getFileType(s), 1)).reduceByKey(new FileTypePartitioner(),_+_)\nBut I can't figure out how to do the same in Python. The closest I can get is to call repartition before reduceByKey like so:\nweblogs.map(lambda s: (getFileType(s), 1)).partitionBy(3,hash_filetype).reduceByKey(lambda v1,v2: v1+v2).collect()\nBut that defeats the purpose, because I'm shuffling twice instead of once, so my performance is worse instead of better.\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #8569 from holdenk/SPARK-9821-pyspark-reduceByKey-should-take-a-custom-partitioner.","distinct":true,"url":"https://api.github.com/repos/dbtsai/spark/commits/1cd67415728e660a90e4dbe136272b5d6b8f1142"},{"sha":"bf20d6c9f9e478a5de24b45bbafd4dd89666c4cf","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-10716] [BUILD] spark-1.5.0-bin-hadoop2.6.tgz file doesn't uncompress on OS X due to hidden file\n\nRemove ._SUCCESS.crc hidden file that may cause problems in distribution tar archive, and is not used\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #8846 from srowen/SPARK-10716.","distinct":true,"url":"https://api.github.com/repos/dbtsai/spark/commits/bf20d6c9f9e478a5de24b45bbafd4dd89666c4cf"},{"sha":"0180b849dbaf191826231eda7dfaaf146a19602b","author":{"email":"a331e1f1d1d90d9284ac06f79235ead859e42933@gmail.com","name":"Jian Feng"},"message":"[SPARK-10577] [PYSPARK] DataFrame hint for broadcast join\n\nhttps://issues.apache.org/jira/browse/SPARK-10577\n\nAuthor: Jian Feng <jzhang.chs@gmail.com>\n\nCloses #8801 from Jianfeng-chs/master.","distinct":true,"url":"https://api.github.com/repos/dbtsai/spark/commits/0180b849dbaf191826231eda7dfaaf146a19602b"},{"sha":"781b21ba2a873ed29394c8dbc74fc700e3e0d17e","author":{"email":"a707759321d733eb98b8d2171fac87d887129d2a@realitymine.com","name":"Ewan Leith"},"message":"[SPARK-10419] [SQL] Adding SQLServer support for datetimeoffset types to JdbcDialects\n\nReading from Microsoft SQL Server over jdbc fails when the table contains datetimeoffset types.\n\nThis patch registers a SQLServer JDBC Dialect that maps datetimeoffset to a String, as Microsoft suggest.\n\nAuthor: Ewan Leith <ewan.leith@realitymine.com>\n\nCloses #8575 from realitymine-coordinator/sqlserver.","distinct":true,"url":"https://api.github.com/repos/dbtsai/spark/commits/781b21ba2a873ed29394c8dbc74fc700e3e0d17e"},{"sha":"1fcefef06950e2f03477282368ca835bbf40ff24","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-10446][SQL] Support to specify join type when calling join with usingColumns\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-10446\n\nCurrently the method `join(right: DataFrame, usingColumns: Seq[String])` only supports inner join. It is more convenient to have it support other join types.\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #8600 from viirya/usingcolumns_df.","distinct":true,"url":"https://api.github.com/repos/dbtsai/spark/commits/1fcefef06950e2f03477282368ca835bbf40ff24"},{"sha":"f24316e6d928c263cbf3872edd97982059c3db22","author":{"email":"1971e0e68de6a19632b65e0c958972b4f2981bb4@in.ibm.com","name":"Madhusudanan Kandasamy"},"message":"[SPARK-10458] [SPARK CORE] Added isStopped() method in SparkContext\n\nAdded isStopped() method in SparkContext\n\nAuthor: Madhusudanan Kandasamy <madhusudanan@in.ibm.com>\n\nCloses #8749 from kmadhugit/SPARK-10458.","distinct":true,"url":"https://api.github.com/repos/dbtsai/spark/commits/f24316e6d928c263cbf3872edd97982059c3db22"},{"sha":"fd61b004877ba4d51c95cd0e08f53bffdf106395","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[Minor] style fix for previous commit f24316e","distinct":true,"url":"https://api.github.com/repos/dbtsai/spark/commits/fd61b004877ba4d51c95cd0e08f53bffdf106395"},{"sha":"4da32bc0e747fefe847bffe493785d4d16069c04","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-8567] [SQL] Increase the timeout of o.a.s.sql.hive.HiveSparkSubmitSuite to 5 minutes.\n\nhttps://issues.apache.org/jira/browse/SPARK-8567\n\nLooks like \"SPARK-8368: includes jars passed in through --jars\" is pretty flaky now. Based on some history runs, the time spent on a successful run may be from 1.5 minutes to almost 3 minutes. Let's try to increase the timeout and see if we can fix this test.\n\nhttps://amplab.cs.berkeley.edu/jenkins/job/Spark-1.5-SBT/AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.0,label=spark-test/385/testReport/junit/org.apache.spark.sql.hive/HiveSparkSubmitSuite/SPARK_8368__includes_jars_passed_in_through___jars/history/?start=25\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #8850 from yhuai/SPARK-8567-anotherTry.","distinct":true,"url":"https://api.github.com/repos/dbtsai/spark/commits/4da32bc0e747fefe847bffe493785d4d16069c04"},{"sha":"f3b727c801408b1cd50e5d9463f2fe0fce654a16","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SQL] [MINOR] map -> foreach.\n\nDataFrame.explain should use foreach to print the explain content.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #8862 from rxin/map-foreach.","distinct":true,"url":"https://api.github.com/repos/dbtsai/spark/commits/f3b727c801408b1cd50e5d9463f2fe0fce654a16"},{"sha":"0bd0e5bed2176b119b3ada590993e153757ea09b","author":{"email":"7ba0da326b36621a0fa584ce084ec977fe1343f7@gmail.com","name":"Akash Mishra"},"message":"[SPARK-10695] [DOCUMENTATION] [MESOS] Fixing incorrect value informati…\n\n…on for spark.mesos.constraints parameter.\n\nAuthor: Akash Mishra <akash.mishra20@gmail.com>\n\nCloses #8816 from SleepyThread/constraint-fix.","distinct":true,"url":"https://api.github.com/repos/dbtsai/spark/commits/0bd0e5bed2176b119b3ada590993e153757ea09b"}]}