{"push_id":758266962,"size":56,"distinct_size":56,"ref":"refs/heads/master","head":"2b44bf2a38b8071b5c71b228945cebf1a999d684","before":"76c79c2cfd69deaf3604b9fcc1520fad5bfa6151","commits":[{"sha":"741a29f98945538a475579ccc974cd42c1613be4","author":{"email":"22a0dda5f6015b8ee84a30ff6553d35a91fe8732@gmail.com","name":"Timothy Chen"},"message":"[SPARK-9575] [MESOS] Add docuemntation around Mesos shuffle service.\n\nandrewor14\n\nAuthor: Timothy Chen <tnachen@gmail.com>\n\nCloses #7907 from tnachen/mesos_shuffle.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/741a29f98945538a475579ccc974cd42c1613be4"},{"sha":"9d0822455ddc8d765440d58c463367a4d67ef456","author":{"email":"348ff379544b45aa3f16e53a543620141423a0f6@gmail.com","name":"Yijie Shen"},"message":"[SPARK-9182] [SQL] Filters are not passed through to jdbc source\n\nThis PR fixes unable to push filter down to JDBC source caused by `Cast` during pattern matching.\n\nWhile we are comparing columns of different type, there's a big chance we need a cast on the column, therefore not match the pattern directly on Attribute and would fail to push down.\n\nAuthor: Yijie Shen <henry.yijieshen@gmail.com>\n\nCloses #8049 from yjshen/jdbc_pushdown.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/9d0822455ddc8d765440d58c463367a4d67ef456"},{"sha":"3ecb3794302dc12d0989f8d725483b2cc37762cf","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-9407] [SQL] Relaxes Parquet ValidTypeMap to allow ENUM predicates to be pushed down\n\nThis PR adds a hacky workaround for PARQUET-201, and should be removed once we upgrade to parquet-mr 1.8.1 or higher versions.\n\nIn Parquet, not all types of columns can be used for filter push-down optimization.  The set of valid column types is controlled by `ValidTypeMap`.  Unfortunately, in parquet-mr 1.7.0 and prior versions, this limitation is too strict, and doesn't allow `BINARY (ENUM)` columns to be pushed down.  On the other hand, `BINARY (ENUM)` is commonly seen in Parquet files written by libraries like `parquet-avro`.\n\nThis restriction is problematic for Spark SQL, because Spark SQL doesn't have a type that maps to Parquet `BINARY (ENUM)` directly, and always converts `BINARY (ENUM)` to Catalyst `StringType`.  Thus, a predicate involving a `BINARY (ENUM)` is recognized as one involving a string field instead and can be pushed down by the query optimizer.  Such predicates are actually perfectly legal except that it fails the `ValidTypeMap` check.\n\nThe workaround added here is relaxing `ValidTypeMap` to include `BINARY (ENUM)`.  I also took the chance to simplify `ParquetCompatibilityTest` a little bit when adding regression test.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #8107 from liancheng/spark-9407/parquet-enum-filter-push-down.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/3ecb3794302dc12d0989f8d725483b2cc37762cf"},{"sha":"2e680668f7b6fc158aa068aedd19c1878ecf759e","author":{"email":"96835dd8bfa718bd6447ccc87af89ae1675daeca@cloudera.com","name":"Tom White"},"message":"[SPARK-8625] [CORE] Propagate user exceptions in tasks back to driver\n\nThis allows clients to retrieve the original exception from the\ncause field of the SparkException that is thrown by the driver.\nIf the original exception is not in fact Serializable then it will\nnot be returned, but the message and stacktrace will be. (All Java\nThrowables implement the Serializable interface, but this is no\nguarantee that a particular implementation can actually be\nserialized.)\n\nAuthor: Tom White <tom@cloudera.com>\n\nCloses #7014 from tomwhite/propagate-user-exceptions.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/2e680668f7b6fc158aa068aedd19c1878ecf759e"},{"sha":"be5d1912076c2ffd21ec88611e53d3b3c59b7ecc","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-9795] Dynamic allocation: avoid double counting when killing same executor twice\n\nThis is based on KaiXinXiaoLei's changes in #7716.\n\nThe issue is that when someone calls `sc.killExecutor(\"1\")` on the same executor twice quickly, then the executor target will be adjusted downwards by 2 instead of 1 even though we're only actually killing one executor. In certain cases where we don't adjust the target back upwards quickly, we'll end up with jobs hanging.\n\nThis is a common danger because there are many places where this is called:\n- `HeartbeatReceiver` kills an executor that has not been sending heartbeats\n- `ExecutorAllocationManager` kills an executor that has been idle\n- The user code might call this, which may interfere with the previous callers\n\nWhile it's not clear whether this fixes SPARK-9745, fixing this potential race condition seems like a strict improvement. I've added a regression test to illustrate the issue.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #8078 from andrewor14/da-double-kill.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/be5d1912076c2ffd21ec88611e53d3b3c59b7ecc"},{"sha":"66d87c1d76bea2b81993156ac1fa7dad6c312ebf","author":{"email":"66332c93c294b372ab1bdd5acb9b89a538e2d2f2@gmail.com","name":"Yuhao Yang"},"message":"[SPARK-7583] [MLLIB] User guide update for RegexTokenizer\n\njira: https://issues.apache.org/jira/browse/SPARK-7583\n\nUser guide update for RegexTokenizer\n\nAuthor: Yuhao Yang <hhbyyh@gmail.com>\n\nCloses #7828 from hhbyyh/regexTokenizerDoc.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/66d87c1d76bea2b81993156ac1fa7dad6c312ebf"},{"sha":"e0110792ef71ebfd3727b970346a2e13695990a4","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-9747] [SQL] Avoid starving an unsafe operator in aggregation\n\nThis is the sister patch to #8011, but for aggregation.\n\nIn a nutshell: create the `TungstenAggregationIterator` before computing the parent partition. Internally this creates a `BytesToBytesMap` which acquires a page in the constructor as of this patch. This ensures that the aggregation operator is not starved since we reserve at least 1 page in advance.\n\nrxin yhuai\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #8038 from andrewor14/unsafe-starve-memory-agg.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/e0110792ef71ebfd3727b970346a2e13695990a4"},{"sha":"57ec27dd7784ce15a2ece8a6c8ac7bd5fd25aea2","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-9804] [HIVE] Use correct value for isSrcLocal parameter.\n\nIf the correct parameter is not provided, Hive will run into an error\nbecause it calls methods that are specific to the local filesystem to\ncopy the data.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #8086 from vanzin/SPARK-9804.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/57ec27dd7784ce15a2ece8a6c8ac7bd5fd25aea2"},{"sha":"70fe558867ccb4bcff6ec673438b03608bb02252","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-9847] [ML] Modified copyValues to distinguish between default, explicit param values\n\nFrom JIRA: Currently, Params.copyValues copies default parameter values to the paramMap of the target instance, rather than the defaultParamMap. It should copy to the defaultParamMap because explicitly setting a parameter can change the semantics.\nThis issue arose in SPARK-9789, where 2 params \"threshold\" and \"thresholds\" for LogisticRegression can have mutually exclusive values. If thresholds is set, then fit() will copy the default value of threshold as well, easily resulting in inconsistent settings for the 2 params.\n\nCC: mengxr\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #8115 from jkbradley/copyvalues-fix.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/70fe558867ccb4bcff6ec673438b03608bb02252"},{"sha":"60103ecd3d9c92709a5878be7ebd57012813ab48","author":{"email":"3c2365aa085787349a5327558db844b55eabde30@brennanashton.com","name":"Brennan Ashton"},"message":"[SPARK-9726] [PYTHON] PySpark DF join no longer accepts on=None\n\nrxin\n\nFirst pull request for Spark so let me know if I am missing anything\nThe contribution is my original work and I license the work to the project under the project's open source license.\n\nAuthor: Brennan Ashton <bashton@brennanashton.com>\n\nCloses #8016 from btashton/patch-1.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/60103ecd3d9c92709a5878be7ebd57012813ab48"},{"sha":"762bacc16ac5e74c8b05a7c1e3e367d1d1633cef","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-9766] [ML] [PySpark] check and add miss docs for PySpark ML\n\nCheck and add miss docs for PySpark ML (this issue only check miss docs for o.a.s.ml not o.a.s.mllib).\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8059 from yanboliang/SPARK-9766.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/762bacc16ac5e74c8b05a7c1e3e367d1d1633cef"},{"sha":"551def5d6972440365bd7436d484a67138d9a8f3","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-9789] [ML] Added logreg threshold param back\n\nReinstated LogisticRegression.threshold Param for binary compatibility.  Param thresholds overrides threshold, if set.\n\nCC: mengxr dbtsai feynmanliang\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #8079 from jkbradley/logreg-reinstate-threshold.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/551def5d6972440365bd7436d484a67138d9a8f3"},{"sha":"6f60298b1d7aa97268a42eca1e3b4851a7e88cb5","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-8967] [DOC] add Since annotation\n\nAdd `Since` as a Scala annotation. The benefit is that we can use it without having explicit JavaDoc. This is useful for inherited methods. The limitation is that is doesn't show up in the generated Java API documentation. This might be fixed by modifying genjavadoc. I think we could leave it as a TODO.\n\nThis is how the generated Scala doc looks:\n\n`since` JavaDoc tag:\n\n![screen shot 2015-08-11 at 10 00 37 pm](https://cloud.githubusercontent.com/assets/829644/9230761/fa72865c-40d8-11e5-807e-0f3c815c5acd.png)\n\n`Since` annotation:\n\n![screen shot 2015-08-11 at 10 00 28 pm](https://cloud.githubusercontent.com/assets/829644/9230764/0041d7f4-40d9-11e5-8124-c3f3e5d5b31f.png)\n\nrxin\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #8131 from mengxr/SPARK-8967.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/6f60298b1d7aa97268a42eca1e3b4851a7e88cb5"},{"sha":"a17384fa343628cec44437da5b80b9403ecd5838","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9907] [SQL] Python crc32 is mistakenly calling md5\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #8138 from rxin/SPARK-9907.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/a17384fa343628cec44437da5b80b9403ecd5838"},{"sha":"738f353988dbf02704bd63f5e35d94402c59ed79","author":{"email":"533a1a34da2a9dba987226baf3d4809f0bdd669f@cloudera.com","name":"Niranjan Padmanabhan"},"message":"[SPARK-9092] Fixed incompatibility when both num-executors and dynamic...\n\n… allocation are set. Now, dynamic allocation is set to false when num-executors is explicitly specified as an argument. Consequently, executorAllocationManager in not initialized in the SparkContext.\n\nAuthor: Niranjan Padmanabhan <niranjan.padmanabhan@cloudera.com>\n\nCloses #7657 from neurons/SPARK-9092.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/738f353988dbf02704bd63f5e35d94402c59ed79"},{"sha":"ab7e721cfec63155641e81e72b4ad43cf6a7d4c7","author":{"email":"6f83a1a7b2ebc528df4d924c90a7c381fba94303@gmail.com","name":"Michel Lemay"},"message":"[SPARK-9826] [CORE] Fix cannot use custom classes in log4j.properties\n\nRefactor Utils class and create ShutdownHookManager.\n\nNOTE: Wasn't able to run /dev/run-tests on windows machine.\nManual tests were conducted locally using custom log4j.properties file with Redis appender and logstash formatter (bundled in the fat-jar submitted to spark)\n\nex:\nlog4j.rootCategory=WARN,console,redis\nlog4j.appender.console=org.apache.log4j.ConsoleAppender\nlog4j.appender.console.target=System.err\nlog4j.appender.console.layout=org.apache.log4j.PatternLayout\nlog4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n\n\nlog4j.logger.org.eclipse.jetty=WARN\nlog4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR\nlog4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO\nlog4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO\nlog4j.logger.org.apache.spark.graphx.Pregel=INFO\n\nlog4j.appender.redis=com.ryantenney.log4j.FailoverRedisAppender\nlog4j.appender.redis.endpoints=hostname:port\nlog4j.appender.redis.key=mykey\nlog4j.appender.redis.alwaysBatch=false\nlog4j.appender.redis.layout=net.logstash.log4j.JSONEventLayoutV1\n\nAuthor: michellemay <mlemay@gmail.com>\n\nCloses #8109 from michellemay/SPARK-9826.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/ab7e721cfec63155641e81e72b4ad43cf6a7d4c7"},{"sha":"7035d880a0cf06910c19b4afd49645124c620f14","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-9894] [SQL] Json writer should handle MapData.\n\nhttps://issues.apache.org/jira/browse/SPARK-9894\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #8137 from yhuai/jsonMapData.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/7035d880a0cf06910c19b4afd49645124c620f14"},{"sha":"caa14d9dc9e2eb1102052b22445b63b0e004e3c7","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-9913] [MLLIB] LDAUtils should be private\n\nfeynmanliang\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #8142 from mengxr/SPARK-9913.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/caa14d9dc9e2eb1102052b22445b63b0e004e3c7"},{"sha":"6e409bc1357f49de2efdfc4226d074b943fb1153","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-9909] [ML] [TRIVIAL] move weightCol to shared params\n\nAs per the TODO move weightCol to Shared Params.\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #8144 from holdenk/SPARK-9909-move-weightCol-toSharedParams.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/6e409bc1357f49de2efdfc4226d074b943fb1153"},{"sha":"e6aef55766d0e2a48e0f9cb6eda0e31a71b962f3","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-9912] [MLLIB] QRDecomposition should use QType and RType for type names instead of UType and VType\n\nhhbyyh\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #8140 from mengxr/SPARK-9912.","distinct":true,"url":"https://api.github.com/repos/sachintyagi22/spark/commits/e6aef55766d0e2a48e0f9cb6eda0e31a71b962f3"}]}