{"push_id":779103319,"size":138,"distinct_size":1,"ref":"refs/heads/master","head":"46d6592012702e21e7d5ab39ef99c10329bb3be3","before":"99e638842fd285c24fd703285e3b18dcfec86186","commits":[{"sha":"e0110792ef71ebfd3727b970346a2e13695990a4","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-9747] [SQL] Avoid starving an unsafe operator in aggregation\n\nThis is the sister patch to #8011, but for aggregation.\n\nIn a nutshell: create the `TungstenAggregationIterator` before computing the parent partition. Internally this creates a `BytesToBytesMap` which acquires a page in the constructor as of this patch. This ensures that the aggregation operator is not starved since we reserve at least 1 page in advance.\n\nrxin yhuai\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #8038 from andrewor14/unsafe-starve-memory-agg.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/e0110792ef71ebfd3727b970346a2e13695990a4"},{"sha":"57ec27dd7784ce15a2ece8a6c8ac7bd5fd25aea2","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-9804] [HIVE] Use correct value for isSrcLocal parameter.\n\nIf the correct parameter is not provided, Hive will run into an error\nbecause it calls methods that are specific to the local filesystem to\ncopy the data.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #8086 from vanzin/SPARK-9804.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/57ec27dd7784ce15a2ece8a6c8ac7bd5fd25aea2"},{"sha":"70fe558867ccb4bcff6ec673438b03608bb02252","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-9847] [ML] Modified copyValues to distinguish between default, explicit param values\n\nFrom JIRA: Currently, Params.copyValues copies default parameter values to the paramMap of the target instance, rather than the defaultParamMap. It should copy to the defaultParamMap because explicitly setting a parameter can change the semantics.\nThis issue arose in SPARK-9789, where 2 params \"threshold\" and \"thresholds\" for LogisticRegression can have mutually exclusive values. If thresholds is set, then fit() will copy the default value of threshold as well, easily resulting in inconsistent settings for the 2 params.\n\nCC: mengxr\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #8115 from jkbradley/copyvalues-fix.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/70fe558867ccb4bcff6ec673438b03608bb02252"},{"sha":"60103ecd3d9c92709a5878be7ebd57012813ab48","author":{"email":"3c2365aa085787349a5327558db844b55eabde30@brennanashton.com","name":"Brennan Ashton"},"message":"[SPARK-9726] [PYTHON] PySpark DF join no longer accepts on=None\n\nrxin\n\nFirst pull request for Spark so let me know if I am missing anything\nThe contribution is my original work and I license the work to the project under the project's open source license.\n\nAuthor: Brennan Ashton <bashton@brennanashton.com>\n\nCloses #8016 from btashton/patch-1.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/60103ecd3d9c92709a5878be7ebd57012813ab48"},{"sha":"762bacc16ac5e74c8b05a7c1e3e367d1d1633cef","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-9766] [ML] [PySpark] check and add miss docs for PySpark ML\n\nCheck and add miss docs for PySpark ML (this issue only check miss docs for o.a.s.ml not o.a.s.mllib).\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8059 from yanboliang/SPARK-9766.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/762bacc16ac5e74c8b05a7c1e3e367d1d1633cef"},{"sha":"551def5d6972440365bd7436d484a67138d9a8f3","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-9789] [ML] Added logreg threshold param back\n\nReinstated LogisticRegression.threshold Param for binary compatibility.  Param thresholds overrides threshold, if set.\n\nCC: mengxr dbtsai feynmanliang\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #8079 from jkbradley/logreg-reinstate-threshold.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/551def5d6972440365bd7436d484a67138d9a8f3"},{"sha":"6f60298b1d7aa97268a42eca1e3b4851a7e88cb5","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-8967] [DOC] add Since annotation\n\nAdd `Since` as a Scala annotation. The benefit is that we can use it without having explicit JavaDoc. This is useful for inherited methods. The limitation is that is doesn't show up in the generated Java API documentation. This might be fixed by modifying genjavadoc. I think we could leave it as a TODO.\n\nThis is how the generated Scala doc looks:\n\n`since` JavaDoc tag:\n\n![screen shot 2015-08-11 at 10 00 37 pm](https://cloud.githubusercontent.com/assets/829644/9230761/fa72865c-40d8-11e5-807e-0f3c815c5acd.png)\n\n`Since` annotation:\n\n![screen shot 2015-08-11 at 10 00 28 pm](https://cloud.githubusercontent.com/assets/829644/9230764/0041d7f4-40d9-11e5-8124-c3f3e5d5b31f.png)\n\nrxin\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #8131 from mengxr/SPARK-8967.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/6f60298b1d7aa97268a42eca1e3b4851a7e88cb5"},{"sha":"a17384fa343628cec44437da5b80b9403ecd5838","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9907] [SQL] Python crc32 is mistakenly calling md5\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #8138 from rxin/SPARK-9907.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/a17384fa343628cec44437da5b80b9403ecd5838"},{"sha":"738f353988dbf02704bd63f5e35d94402c59ed79","author":{"email":"533a1a34da2a9dba987226baf3d4809f0bdd669f@cloudera.com","name":"Niranjan Padmanabhan"},"message":"[SPARK-9092] Fixed incompatibility when both num-executors and dynamic...\n\n… allocation are set. Now, dynamic allocation is set to false when num-executors is explicitly specified as an argument. Consequently, executorAllocationManager in not initialized in the SparkContext.\n\nAuthor: Niranjan Padmanabhan <niranjan.padmanabhan@cloudera.com>\n\nCloses #7657 from neurons/SPARK-9092.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/738f353988dbf02704bd63f5e35d94402c59ed79"},{"sha":"ab7e721cfec63155641e81e72b4ad43cf6a7d4c7","author":{"email":"6f83a1a7b2ebc528df4d924c90a7c381fba94303@gmail.com","name":"Michel Lemay"},"message":"[SPARK-9826] [CORE] Fix cannot use custom classes in log4j.properties\n\nRefactor Utils class and create ShutdownHookManager.\n\nNOTE: Wasn't able to run /dev/run-tests on windows machine.\nManual tests were conducted locally using custom log4j.properties file with Redis appender and logstash formatter (bundled in the fat-jar submitted to spark)\n\nex:\nlog4j.rootCategory=WARN,console,redis\nlog4j.appender.console=org.apache.log4j.ConsoleAppender\nlog4j.appender.console.target=System.err\nlog4j.appender.console.layout=org.apache.log4j.PatternLayout\nlog4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n\n\nlog4j.logger.org.eclipse.jetty=WARN\nlog4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR\nlog4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO\nlog4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO\nlog4j.logger.org.apache.spark.graphx.Pregel=INFO\n\nlog4j.appender.redis=com.ryantenney.log4j.FailoverRedisAppender\nlog4j.appender.redis.endpoints=hostname:port\nlog4j.appender.redis.key=mykey\nlog4j.appender.redis.alwaysBatch=false\nlog4j.appender.redis.layout=net.logstash.log4j.JSONEventLayoutV1\n\nAuthor: michellemay <mlemay@gmail.com>\n\nCloses #8109 from michellemay/SPARK-9826.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/ab7e721cfec63155641e81e72b4ad43cf6a7d4c7"},{"sha":"7035d880a0cf06910c19b4afd49645124c620f14","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-9894] [SQL] Json writer should handle MapData.\n\nhttps://issues.apache.org/jira/browse/SPARK-9894\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #8137 from yhuai/jsonMapData.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/7035d880a0cf06910c19b4afd49645124c620f14"},{"sha":"caa14d9dc9e2eb1102052b22445b63b0e004e3c7","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-9913] [MLLIB] LDAUtils should be private\n\nfeynmanliang\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #8142 from mengxr/SPARK-9913.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/caa14d9dc9e2eb1102052b22445b63b0e004e3c7"},{"sha":"6e409bc1357f49de2efdfc4226d074b943fb1153","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-9909] [ML] [TRIVIAL] move weightCol to shared params\n\nAs per the TODO move weightCol to Shared Params.\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #8144 from holdenk/SPARK-9909-move-weightCol-toSharedParams.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/6e409bc1357f49de2efdfc4226d074b943fb1153"},{"sha":"e6aef55766d0e2a48e0f9cb6eda0e31a71b962f3","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-9912] [MLLIB] QRDecomposition should use QType and RType for type names instead of UType and VType\n\nhhbyyh\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #8140 from mengxr/SPARK-9912.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/e6aef55766d0e2a48e0f9cb6eda0e31a71b962f3"},{"sha":"fc1c7fd66e64ccea53b31cd2fbb98bc6d307329c","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-9915] [ML] stopWords should use StringArrayParam\n\nhhbyyh\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #8141 from mengxr/SPARK-9915.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/fc1c7fd66e64ccea53b31cd2fbb98bc6d307329c"},{"sha":"660e6dcff8125b83cc73dbe00c90cbe58744bc66","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-9449] [SQL] Include MetastoreRelation's inputFiles\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #8119 from marmbrus/metastoreInputFiles.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/660e6dcff8125b83cc73dbe00c90cbe58744bc66"},{"sha":"8ce60963cb0928058ef7b6e29ff94eb69d1143af","author":{"email":"c4fe2b1d90ef8f2548c8aeabfa633434316f0305@koeninger.org","name":"cody koeninger"},"message":"[SPARK-9780] [STREAMING] [KAFKA] prevent NPE if KafkaRDD instantiation …\n\n…fails\n\nAuthor: cody koeninger <cody@koeninger.org>\n\nCloses #8133 from koeninger/SPARK-9780 and squashes the following commits:\n\n406259d [cody koeninger] [SPARK-9780][Streaming][Kafka] prevent NPE if KafkaRDD instantiation fails","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/8ce60963cb0928058ef7b6e29ff94eb69d1143af"},{"sha":"0d1d146c220f0d47d0e62b368d5b94d3bd9dd197","author":{"email":"bc6454280780a091e4ce61732d29d654dd50f5a3@qubole.com","name":"Rohit Agarwal"},"message":"[SPARK-9724] [WEB UI] Avoid unnecessary redirects in the Spark Web UI.\n\nAuthor: Rohit Agarwal <rohita@qubole.com>\n\nCloses #8014 from mindprince/SPARK-9724 and squashes the following commits:\n\na7af5ff [Rohit Agarwal] [SPARK-9724] [WEB UI] Inline attachPrefix and attachPrefixForRedirect. Fix logic of attachPrefix\n8a977cd [Rohit Agarwal] [SPARK-9724] [WEB UI] Address review comments: Remove unneeded code, update scaladoc.\nb257844 [Rohit Agarwal] [SPARK-9724] [WEB UI] Avoid unnecessary redirects in the Spark Web UI.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/0d1d146c220f0d47d0e62b368d5b94d3bd9dd197"},{"sha":"f4bc01f1f33a93e6affe5c8a3e33ffbd92d03f38","author":{"email":"bc4a65f227ddcd81bb538c67e810a3e11696e8bf@gmail.com","name":"Yu ISHIKAWA"},"message":"[SPARK-9855] [SPARKR] Add expression functions into SparkR whose params are simple\n\nI added lots of expression functions for SparkR. This PR includes only functions whose params  are only `(Column)` or `(Column, Column)`.  And I think we need to improve how to test those functions. However, it would be better to work on another issue.\n\n## Diff Summary\n\n- Add lots of functions in `functions.R` and their generic in `generic.R`\n- Add aliases for `ceiling` and `sign`\n- Move expression functions from `column.R` to `functions.R`\n- Modify `rdname` from `column` to `functions`\n\nI haven't supported `not` function, because the name has a collesion with `testthat` package. I didn't think of the way  to define it.\n\n## New Supported Functions\n\n```\napproxCountDistinct\nascii\nbase64\nbin\nbitwiseNOT\nceil (alias: ceiling)\ncrc32\ndayofmonth\ndayofyear\nexplode\nfactorial\nhex\nhour\ninitcap\nisNaN\nlast_day\nlength\nlog2\nltrim\nmd5\nminute\nmonth\nnegate\nquarter\nreverse\nround\nrtrim\nsecond\nsha1\nsignum (alias: sign)\nsize\nsoundex\nto_date\ntrim\nunbase64\nunhex\nweekofyear\nyear\n\ndatediff\nlevenshtein\nmonths_between\nnanvl\npmod\n```\n\n## JIRA\n[[SPARK-9855] Add expression functions into SparkR whose params are simple - ASF JIRA](https://issues.apache.org/jira/browse/SPARK-9855)\n\nAuthor: Yu ISHIKAWA <yuu.ishikawa@gmail.com>\n\nCloses #8123 from yu-iskw/SPARK-9855.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/f4bc01f1f33a93e6affe5c8a3e33ffbd92d03f38"},{"sha":"7b13ed27c1296cf76d0946e400f3449c335c8471","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-9870] Disable driver UI and Master REST server in SparkSubmitSuite\n\nI think that we should pass additional configuration flags to disable the driver UI and Master REST server in SparkSubmitSuite and HiveSparkSubmitSuite. This might cut down on port-contention-related flakiness in Jenkins.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #8124 from JoshRosen/disable-ui-in-sparksubmitsuite.","distinct":false,"url":"https://api.github.com/repos/Shopify/spark/commits/7b13ed27c1296cf76d0946e400f3449c335c8471"}]}