{"push_id":746982065,"size":22,"distinct_size":2,"ref":"refs/heads/codegen_in","head":"cf4bf41b4070227f05ac1c21bf81799665c91183","before":"446bbcdcea92e8545a76e29ab03a097e81e865e6","commits":[{"sha":"69f5a7c934ac553ed52c00679b800bcffe83c1d6","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-9528] [ML] Changed RandomForestClassifier to extend ProbabilisticClassifier\n\nRandomForestClassifier now outputs rawPrediction based on tree probabilities, plus probability column computed from normalized rawPrediction.\n\nCC: holdenk\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #7859 from jkbradley/rf-prob and squashes the following commits:\n\n6c28f51 [Joseph K. Bradley] Changed RandomForestClassifier to extend ProbabilisticClassifier","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/69f5a7c934ac553ed52c00679b800bcffe83c1d6"},{"sha":"b41a32718d615b304efba146bf97be0229779b01","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-1855] Local checkpointing\n\nCertain use cases of Spark involve RDDs with long lineages that must be truncated periodically (e.g. GraphX). The existing way of doing it is through `rdd.checkpoint()`, which is expensive because it writes to HDFS. This patch provides an alternative to truncate lineages cheaply *without providing the same level of fault tolerance*.\n\n**Local checkpointing** writes checkpointed data to the local file system through the block manager. It is much faster than replicating to a reliable storage and provides the same semantics as long as executors do not fail. It is accessible through a new operator `rdd.localCheckpoint()` and leaves the old one unchanged. Users may even decide to combine the two and call the reliable one less frequently.\n\nThe bulk of this patch involves refactoring the checkpointing interface to accept custom implementations of checkpointing. [Design doc](https://issues.apache.org/jira/secure/attachment/12741708/SPARK-7292-design.pdf).\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #7279 from andrewor14/local-checkpoint and squashes the following commits:\n\n729600f [Andrew Or] Oops, fix tests\n34bc059 [Andrew Or] Avoid computing all partitions in local checkpoint\ne43bbb6 [Andrew Or] Merge branch 'master' of github.com:apache/spark into local-checkpoint\n3be5aea [Andrew Or] Address comments\nbf846a6 [Andrew Or] Merge branch 'master' of github.com:apache/spark into local-checkpoint\nab003a3 [Andrew Or] Fix compile\nc2e111b [Andrew Or] Address comments\n33f167a [Andrew Or] Merge branch 'master' of github.com:apache/spark into local-checkpoint\ne908a42 [Andrew Or] Fix tests\nf5be0f3 [Andrew Or] Use MEMORY_AND_DISK as the default local checkpoint level\na92657d [Andrew Or] Update a few comments\ne58e3e3 [Andrew Or] Merge branch 'master' of github.com:apache/spark into local-checkpoint\n4eb6eb1 [Andrew Or] Merge branch 'master' of github.com:apache/spark into local-checkpoint\n1bbe154 [Andrew Or] Simplify LocalCheckpointRDD\n48a9996 [Andrew Or] Avoid traversing dependency tree + rewrite tests\n62aba3f [Andrew Or] Merge branch 'master' of github.com:apache/spark into local-checkpoint\ndb70dc2 [Andrew Or] Express local checkpointing through caching the original RDD\n87d43c6 [Andrew Or] Merge branch 'master' of github.com:apache/spark into local-checkpoint\nc449b38 [Andrew Or] Fix style\n4a182f3 [Andrew Or] Add fine-grained tests for local checkpointing\n53b363b [Andrew Or] Rename a few more awkwardly named methods (minor)\ne4cf071 [Andrew Or] Simplify LocalCheckpointRDD + docs + clean ups\n4880deb [Andrew Or] Fix style\nd096c67 [Andrew Or] Fix mima\n172cb66 [Andrew Or] Fix mima?\ne53d964 [Andrew Or] Fix style\n56831c5 [Andrew Or] Add a few warnings and clear exception messages\n2e59646 [Andrew Or] Add local checkpoint clean up tests\n4dbbab1 [Andrew Or] Refactor CheckpointSuite to test local checkpointing\n4514dc9 [Andrew Or] Clean local checkpoint files through RDD cleanups\n0477eec [Andrew Or] Rename a few methods with awkward names (minor)\n2e902e5 [Andrew Or] First implementation of local checkpointing\n8447454 [Andrew Or] Fix tests\n4ac1896 [Andrew Or] Refactor checkpoint interface for modularity","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/b41a32718d615b304efba146bf97be0229779b01"},{"sha":"dfe7bd168d9bcf8c53f993f459ab473d893457b0","author":{"email":"fe0966a084bd02ed13ddc43bf288232282ff6a11@cloudera.com","name":"Joseph Batchik"},"message":"[SPARK-9511] [SQL] Fixed Table Name Parsing\n\nThe issue was that the tokenizer was parsing \"1one\" into the numeric 1 using the code on line 110. I added another case to accept strings that start with a number and then have a letter somewhere else in it as well.\n\nAuthor: Joseph Batchik <joseph.batchik@cloudera.com>\n\nCloses #7844 from JDrit/parse_error and squashes the following commits:\n\nb8ca12f [Joseph Batchik] fixed parsing issue by adding another case","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/dfe7bd168d9bcf8c53f993f459ab473d893457b0"},{"sha":"7a9d09f0bb472a1671d3457e1f7108f4c2eb4121","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SQL][minor] Simplify UnsafeRow.calculateBitSetWidthInBytes.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #7897 from rxin/calculateBitSetWidthInBytes and squashes the following commits:\n\n2e73b3a [Reynold Xin] [SQL][minor] Simplify UnsafeRow.calculateBitSetWidthInBytes.","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/7a9d09f0bb472a1671d3457e1f7108f4c2eb4121"},{"sha":"703e44bff19f4c394f6f9bff1ce9152cdc68c51e","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-9554] [SQL] Enables in-memory partition pruning by default\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #7895 from liancheng/spark-9554/enable-in-memory-partition-pruning and squashes the following commits:\n\n67c403e [Cheng Lian] Enables in-memory partition pruning by default","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/703e44bff19f4c394f6f9bff1ce9152cdc68c51e"},{"sha":"ff9169a002f1b75231fd25b7d04157a912503038","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-5133] [ML] Added featureImportance to RandomForestClassifier and Regressor\n\nAdded featureImportance to RandomForestClassifier and Regressor.\n\nThis follows the scikit-learn implementation here: [https://github.com/scikit-learn/scikit-learn/blob/a95203b249c1cf392f86d001ad999e29b2392739/sklearn/tree/_tree.pyx#L3341]\n\nCC: yanboliang  Would you mind taking a look?  Thanks!\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\nAuthor: Feynman Liang <fliang@databricks.com>\n\nCloses #7838 from jkbradley/dt-feature-importance and squashes the following commits:\n\n72a167a [Joseph K. Bradley] fixed unit test\n86cea5f [Joseph K. Bradley] Modified RF featuresImportances to return Vector instead of Map\n5aa74f0 [Joseph K. Bradley] finally fixed unit test for real\n33df5db [Joseph K. Bradley] fix unit test\n42a2d3b [Joseph K. Bradley] fix unit test\nfe94e72 [Joseph K. Bradley] modified feature importance unit tests\ncc693ee [Feynman Liang] Add classifier tests\n79a6f87 [Feynman Liang] Compare dense vectors in test\n21d01fc [Feynman Liang] Added failing SKLearn test\nac0b254 [Joseph K. Bradley] Added featureImportance to RandomForestClassifier/Regressor.  Need to add unit tests","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/ff9169a002f1b75231fd25b7d04157a912503038"},{"sha":"ba1c4e138de2ea84b55def4eed2bd363e60aea4d","author":{"email":"7d45abe39fd242ee2f6575da28f13d66568418b3@oss.nttdata.co.jp","name":"Kousuke Saruta"},"message":"[SPARK-9558][DOCS]Update docs to follow the increase of memory defaults.\n\nNow the memory defaults of master and slave in Standalone mode and History Server is 1g, not 512m. So let's update docs.\n\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #7896 from sarutak/update-doc-for-daemon-memory and squashes the following commits:\n\na77626c [Kousuke Saruta] Fix docs to follow the update of increase of memory defaults","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/ba1c4e138de2ea84b55def4eed2bd363e60aea4d"},{"sha":"8ca287ebbd58985a568341b08040d0efa9d3641a","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-9191] [ML] [Doc] Add ml.PCA user guide and code examples\n\nAdd ml.PCA user guide document and code examples for Scala/Java/Python.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #7522 from yanboliang/ml-pca-md and squashes the following commits:\n\n60dec05 [Yanbo Liang] address comments\nf992abe [Yanbo Liang] Add ml.PCA doc and examples","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/8ca287ebbd58985a568341b08040d0efa9d3641a"},{"sha":"e4765a46833baff1dd7465c4cf50e947de7e8f21","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-9544] [MLLIB] add Python API for RFormula\n\nAdd Python API for RFormula. Similar to other feature transformers in Python. This is just a thin wrapper over the Scala implementation. ericl MechCoder\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #7879 from mengxr/SPARK-9544 and squashes the following commits:\n\n3d5ff03 [Xiangrui Meng] add an doctest for . and -\n5e969a5 [Xiangrui Meng] fix pydoc\n1cd41f8 [Xiangrui Meng] organize imports\n3c18b10 [Xiangrui Meng] add Python API for RFormula","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/e4765a46833baff1dd7465c4cf50e947de7e8f21"},{"sha":"702aa9d7fb16c98a50e046edfd76b8a7861d0391","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-8735] [SQL] Expose memory usage for shuffles, joins and aggregations\n\nThis patch exposes the memory used by internal data structures on the SparkUI. This tracks memory used by all spilling operations and SQL operators backed by Tungsten, e.g. `BroadcastHashJoin`, `ExternalSort`, `GeneratedAggregate` etc. The metric exposed is \"peak execution memory\", which broadly refers to the peak in-memory sizes of each of these data structure.\n\nA separate patch will extend this by linking the new information to the SQL operators themselves.\n\n<img width=\"950\" alt=\"screen shot 2015-07-29 at 7 43 17 pm\" src=\"https://cloud.githubusercontent.com/assets/2133137/8974776/b90fc980-362a-11e5-9e2b-842da75b1641.png\">\n<img width=\"802\" alt=\"screen shot 2015-07-29 at 7 43 05 pm\" src=\"https://cloud.githubusercontent.com/assets/2133137/8974777/baa76492-362a-11e5-9b77-e364a6a6b64e.png\">\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/apache/spark/7770)\n<!-- Reviewable:end -->\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #7770 from andrewor14/expose-memory-metrics and squashes the following commits:\n\n9abecb9 [Andrew Or] Merge branch 'master' of github.com:apache/spark into expose-memory-metrics\nf5b0d68 [Andrew Or] Merge branch 'master' of github.com:apache/spark into expose-memory-metrics\nd7df332 [Andrew Or] Merge branch 'master' of github.com:apache/spark into expose-memory-metrics\n8eefbc5 [Andrew Or] Fix non-failing tests\n9de2a12 [Andrew Or] Fix tests due to another logical merge conflict\n876bfa4 [Andrew Or] Fix failing test after logical merge conflict\n361a359 [Andrew Or] Merge branch 'master' of github.com:apache/spark into expose-memory-metrics\n40b4802 [Andrew Or] Fix style?\nd0fef87 [Andrew Or] Fix tests?\nb3b92f6 [Andrew Or] Address comments\n0625d73 [Andrew Or] Merge branch 'master' of github.com:apache/spark into expose-memory-metrics\nc00a197 [Andrew Or] Fix potential NPEs\n10da1cd [Andrew Or] Fix compile\n17f4c2d [Andrew Or] Fix compile?\na87b4d0 [Andrew Or] Fix compile?\nd70874d [Andrew Or] Fix test compile + address comments\n2840b7d [Andrew Or] Merge branch 'master' of github.com:apache/spark into expose-memory-metrics\n6aa2f7a [Andrew Or] Merge branch 'master' of github.com:apache/spark into expose-memory-metrics\nb889a68 [Andrew Or] Minor changes: comments, spacing, style\n663a303 [Andrew Or] UnsafeShuffleWriter: update peak memory before close\nd090a94 [Andrew Or] Fix style\n2480d84 [Andrew Or] Expand test coverage\n5f1235b [Andrew Or] Merge branch 'master' of github.com:apache/spark into expose-memory-metrics\n1ecf678 [Andrew Or] Minor changes: comments, style, unused imports\n0b6926c [Andrew Or] Oops\n111a05e [Andrew Or] Merge branch 'master' of github.com:apache/spark into expose-memory-metrics\na7a39a5 [Andrew Or] Strengthen presence check for accumulator\na919eb7 [Andrew Or] Add tests for unsafe shuffle writer\n23c845d [Andrew Or] Add tests for SQL operators\na757550 [Andrew Or] Address comments\nb5c51c1 [Andrew Or] Re-enable test in JavaAPISuite\n5107691 [Andrew Or] Add tests for internal accumulators\n59231e4 [Andrew Or] Fix tests\n9528d09 [Andrew Or] Merge branch 'master' of github.com:apache/spark into expose-memory-metrics\n5b5e6f3 [Andrew Or] Add peak execution memory to summary table + tooltip\n92b4b6b [Andrew Or] Display peak execution memory on the UI\neee5437 [Andrew Or] Merge branch 'master' of github.com:apache/spark into expose-memory-metrics\nd9b9015 [Andrew Or] Track execution memory in unsafe shuffles\n770ee54 [Andrew Or] Track execution memory in broadcast joins\n9c605a4 [Andrew Or] Track execution memory in GeneratedAggregate\n9e824f2 [Andrew Or] Add back execution memory tracking for *ExternalSort\n4ef4cb1 [Andrew Or] Merge branch 'master' of github.com:apache/spark into expose-memory-metrics\ne6c3e2f [Andrew Or] Move internal accumulators creation to Stage\na417592 [Andrew Or] Expose memory metrics in UnsafeExternalSorter\n3c4f042 [Andrew Or] Track memory usage in ExternalAppendOnlyMap / ExternalSorter\nbd7ab3f [Andrew Or] Add internal accumulators to TaskContext","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/702aa9d7fb16c98a50e046edfd76b8a7861d0391"},{"sha":"b2e4b85d2db0320e9cbfaf5a5542f749f1f11cf4","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Revert \"[SPARK-9372] [SQL] Filter nulls in join keys\"\n\nThis reverts commit 687c8c37150f4c93f8e57d86bb56321a4891286b.","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/b2e4b85d2db0320e9cbfaf5a5542f749f1f11cf4"},{"sha":"a2409d1c8e8ddec04b529ac6f6a12b5993f0eeda","author":{"email":"ec969b9b5c2f2a49a0d9175cec61f09554863a3d@hortonworks.com","name":"Steve Loughran"},"message":"[SPARK-8064] [SQL] Build against Hive 1.2.1\n\nCherry picked the parts of the initial SPARK-8064 WiP branch needed to get sql/hive to compile against hive 1.2.1. That's the ASF release packaged under org.apache.hive, not any fork.\n\nTests not run yet: that's what the machines are for\n\nAuthor: Steve Loughran <stevel@hortonworks.com>\nAuthor: Cheng Lian <lian@databricks.com>\nAuthor: Michael Armbrust <michael@databricks.com>\nAuthor: Patrick Wendell <patrick@databricks.com>\n\nCloses #7191 from steveloughran/stevel/feature/SPARK-8064-hive-1.2-002 and squashes the following commits:\n\n7556d85 [Cheng Lian] Updates .q files and corresponding golden files\nef4af62 [Steve Loughran] Merge commit '6a92bb09f46a04d6cd8c41bdba3ecb727ebb9030' into stevel/feature/SPARK-8064-hive-1.2-002\n6a92bb0 [Cheng Lian] Overrides HiveConf time vars\ndcbb391 [Cheng Lian] Adds com.twitter:parquet-hadoop-bundle:1.6.0 for Hive Parquet SerDe\n0bbe475 [Steve Loughran] SPARK-8064 scalastyle rejects the standard Hadoop ASF license header...\nfdf759b [Steve Loughran] SPARK-8064 classpath dependency suite to be in sync with shading in final (?) hive-exec spark\n7a6c727 [Steve Loughran] SPARK-8064 switch to second staging repo of the spark-hive artifacts. This one has the protobuf-shaded hive-exec jar\n376c003 [Steve Loughran] SPARK-8064 purge duplicate protobuf declaration\n2c74697 [Steve Loughran] SPARK-8064 switch to the protobuf shaded hive-exec jar with tests to chase it down\ncc44020 [Steve Loughran] SPARK-8064 remove hadoop.version from runtest.py, as profile will fix that automatically.\n6901fa9 [Steve Loughran] SPARK-8064 explicit protobuf import\nda310dc [Michael Armbrust] Fixes for Hive tests.\na775a75 [Steve Loughran] SPARK-8064 cherry-pick-incomplete\n7404f34 [Patrick Wendell] Add spark-hive staging repo\n832c164 [Steve Loughran] SPARK-8064 try to supress compiler warnings on Complex.java pasted-thrift-code\n312c0d4 [Steve Loughran] SPARK-8064  maven/ivy dependency purge; calcite declaration needed\nfa5ae7b [Steve Loughran] HIVE-8064 fix up hive-thriftserver dependencies and cut back on evicted references in the hive- packages; this keeps mvn and ivy resolution compatible, as the reconciliation policy is \"by hand\"\nc188048 [Steve Loughran] SPARK-8064 manage the Hive depencencies to that -things that aren't needed are excluded -sql/hive built with ivy is in sync with the maven reconciliation policy, rather than latest-first\n4c8be8d [Cheng Lian] WIP: Partial fix for Thrift server and CLI tests\n314eb3c [Steve Loughran] SPARK-8064 deprecation warning  noise in one of the tests\n17b0341 [Steve Loughran] SPARK-8064 IDE-hinted cleanups of Complex.java to reduce compiler warnings. It's all autogenerated code, so still ugly.\nd029b92 [Steve Loughran] SPARK-8064 rely on unescaping to have already taken place, so go straight to map of serde options\n23eca7e [Steve Loughran] HIVE-8064 handle raw and escaped property tokens\n54d9b06 [Steve Loughran] SPARK-8064 fix compilation regression surfacing from rebase\n0b12d5f [Steve Loughran] HIVE-8064 use subset of hive complex type whose types deserialize\nfce73b6 [Steve Loughran] SPARK-8064 poms rely implicitly on the version of kryo chill provides\nfd3aa5d [Steve Loughran] SPARK-8064 version of hive to d/l from ivy is 1.2.1\ndc73ece [Steve Loughran] SPARK-8064 revert to master's determinstic pushdown strategy\nd3c1e4a [Steve Loughran] SPARK-8064 purge UnionType\n051cc21 [Steve Loughran] SPARK-8064 switch to an unshaded version of hive-exec-core, which must have been built with Kryo 2.21. This currently looks for a (locally built) version 1.2.1.spark\n6684c60 [Steve Loughran] SPARK-8064 ignore RTE raised in blocking process.exitValue() call\ne6121e5 [Steve Loughran] SPARK-8064 address review comments\naa43dc6 [Steve Loughran] SPARK-8064  more robust teardown on JavaMetastoreDatasourcesSuite\nf2bff01 [Steve Loughran] SPARK-8064 better takeup of asynchronously caught error text\n8b1ef38 [Steve Loughran] SPARK-8064: on failures executing spark-submit in HiveSparkSubmitSuite, print command line and all logged output.\n5a9ce6b [Steve Loughran] SPARK-8064 add explicit reason for kv split failure, rather than array OOB. *does not address the issue*\n642b63a [Steve Loughran] SPARK-8064 reinstate something cut briefly during rebasing\n97194dc [Steve Loughran] SPARK-8064 add extra logging to the YarnClusterSuite classpath test. There should be no reason why this is failing on jenkins, but as it is (and presumably its CP-related), improve the logging including any exception raised.\n335357f [Steve Loughran] SPARK-8064 fail fast on thrive process spawning tests on exit codes and/or error string patterns seen in log.\n3ed872f [Steve Loughran] SPARK-8064 rename field double to  dbl\nbca55e5 [Steve Loughran] SPARK-8064 missed one of the `date` escapes\n41d6479 [Steve Loughran] SPARK-8064 wrap tests with withTable() calls to avoid table-exists exceptions\n2bc29a4 [Steve Loughran] SPARK-8064 ParquetSuites to escape `date` field name\n1ab9bc4 [Steve Loughran] SPARK-8064 TestHive to use sered2.thrift.test.Complex\nbf3a249 [Steve Loughran] SPARK-8064: more resubmit than fix; tighten startup timeout to 60s. Still no obvious reason why jersey server code in spark-assembly isn't being picked up -it hasn't been shaded\nc829b8f [Steve Loughran] SPARK-8064: reinstate yarn-rm-server dependencies to hive-exec to ensure that jersey server is on classpath on hadoop versions < 2.6\n0b0f738 [Steve Loughran] SPARK-8064: thrift server startup to fail fast on any exception in the main thread\n13abaf1 [Steve Loughran] SPARK-8064 Hive compatibilty tests sin sync with explain/show output from Hive 1.2.1\nd14d5ea [Steve Loughran] SPARK-8064: DATE is now a predicate; you can't use it as a field in select ops\n26eef1c [Steve Loughran] SPARK-8064: HIVE-9039 renamed TOK_UNION => TOK_UNIONALL while adding TOK_UNIONDISTINCT\n3d64523 [Steve Loughran] SPARK-8064 improve diagns on uknown token; fix scalastyle failure\nd0360f6 [Steve Loughran] SPARK-8064: delicate merge in of the branch vanzin/hive-1.1\n1126e5a [Steve Loughran] SPARK-8064: name of unrecognized file format wasn't appearing in error text\n8cb09c4 [Steve Loughran] SPARK-8064: test resilience/assertion improvements. Independent of the rest of the work; can be backported to earlier versions\ndec12cb [Steve Loughran] SPARK-8064: when a CLI suite test fails include the full output text in the raised exception; this ensures that the stdout/stderr is included in jenkins reports, so it becomes possible to diagnose the cause.\n463a670 [Steve Loughran] SPARK-8064 run-tests.py adds a hadoop-2.6 profile, and changes info messages to say \"w/Hive 1.2.1\" in console output\n2531099 [Steve Loughran] SPARK-8064 successful attempt to get rid of pentaho as a transitive dependency of hive-exec\n1d59100 [Steve Loughran] SPARK-8064 (unsuccessful) attempt to get rid of pentaho as a transitive dependency of hive-exec\n75733fc [Steve Loughran] SPARK-8064 change thrift binary startup message to \"Starting ThriftBinaryCLIService on port\"\n3ebc279 [Steve Loughran] SPARK-8064 move strings used to check for http/bin thrift services up into constants\nc80979d [Steve Loughran] SPARK-8064: SparkSQLCLIDriver drops remote mode support. CLISuite Tests pass instead of timing out: undetected regression?\n27e8370 [Steve Loughran] SPARK-8064 fix some style & IDE warnings\n00e50d6 [Steve Loughran] SPARK-8064 stop excluding hive shims from dependency (commented out , for now)\ncb4f142 [Steve Loughran] SPARK-8054 cut pentaho dependency from calcite\nf7aa9cb [Steve Loughran] SPARK-8064 everything compiles with some commenting and moving of classes into a hive package\n6c310b4 [Steve Loughran] SPARK-8064 subclass  Hive ServerOptionsProcessor to make it public again\nf61a675 [Steve Loughran] SPARK-8064 thrift server switched to Hive 1.2.1, though it doesn't compile everywhere\n4890b9d [Steve Loughran] SPARK-8064, build against Hive 1.2.1","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/a2409d1c8e8ddec04b529ac6f6a12b5993f0eeda"},{"sha":"13675c742a71cbdc8324701c3694775ce1dd5c62","author":{"email":"8757c4a255384de9fc2175f17a84f948e7081d30@gmail.com","name":"MechCoder"},"message":"[SPARK-8874] [ML] Add missing methods in Word2Vec\n\nAdd missing methods\n\n1. getVectors\n2. findSynonyms\n\nto W2Vec scala and python API\n\nmengxr\n\nAuthor: MechCoder <manojkumarsivaraj334@gmail.com>\n\nCloses #7263 from MechCoder/missing_methods_w2vec and squashes the following commits:\n\n149d5ca [MechCoder] minor doc\n69d91b7 [MechCoder] [SPARK-8874] [ML] Add missing methods in Word2Vec","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/13675c742a71cbdc8324701c3694775ce1dd5c62"},{"sha":"7abaaad5b169520fbf7299808b2bafde089a16a2","author":{"email":"ad63b47dfac1951bde2b661ba9fbbe123f686bee@cs.berkeley.edu","name":"Shivaram Venkataraman"},"message":"Add a prerequisites section for building docs\n\nThis puts all the install commands that need to be run in one section instead of being spread over many paragraphs\n\ncc rxin\n\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\n\nCloses #7912 from shivaram/docs-setup-readme and squashes the following commits:\n\ncf7a204 [Shivaram Venkataraman] Add a prerequisites section for building docs","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/7abaaad5b169520fbf7299808b2bafde089a16a2"},{"sha":"b79b4f5f2251ed7efeec1f4b26e45a8ea6b85a6a","author":{"email":"0f1b6c21e0a79dff0c9e6fc6e738b2412190e920@us.ibm.com","name":"Matthew Brandyberry"},"message":"[SPARK-9483] Fix UTF8String.getPrefix for big-endian.\n\nPrevious code assumed little-endian.\n\nAuthor: Matthew Brandyberry <mbrandy@us.ibm.com>\n\nCloses #7902 from mtbrandy/SPARK-9483 and squashes the following commits:\n\nec31df8 [Matthew Brandyberry] [SPARK-9483] Changes from review comments.\n17d54c6 [Matthew Brandyberry] [SPARK-9483] Fix UTF8String.getPrefix for big-endian.","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/b79b4f5f2251ed7efeec1f4b26e45a8ea6b85a6a"},{"sha":"1633d0a2612d94151f620c919425026150e69ae1","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":"[SPARK-9263] Added flags to exclude dependencies when using --packages\n\nWhile the functionality is there to exclude packages, there are no flags that allow users to exclude dependencies, in case of dependency conflicts. We should provide users with a flag to add dependency exclusions in case the packages are not resolved properly (or not available due to licensing).\n\nThe flag I added was --packages-exclude, but I'm open on renaming it. I also added property flags in case people would like to use a conf file to provide dependencies, which is possible if there is a long list of dependencies or exclusions.\n\ncc andrewor14 vanzin pwendell\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #7599 from brkyvz/packages-exclusions and squashes the following commits:\n\n636f410 [Burak Yavuz] addressed nits\n6e54ede [Burak Yavuz] is this the culprit\nb5e508e [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into packages-exclusions\n154f5db [Burak Yavuz] addressed initial comments\n1536d7a [Burak Yavuz] Added flags to exclude packages using --packages-exclude","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/1633d0a2612d94151f620c919425026150e69ae1"},{"sha":"3b0e44490aebfba30afc147e4a34a63439d985c6","author":{"email":"fd726277649d3b2e3ee5d332dca6c89739b68aa5@gmail.com","name":"CodingCat"},"message":"[SPARK-8416] highlight and topping the executor threads in thread dumping page\n\nhttps://issues.apache.org/jira/browse/SPARK-8416\n\nTo facilitate debugging, I made this patch with three changes:\n\n* render the executor-thread and non executor-thread entries with different background colors\n\n* put the executor threads on the top of the list\n\n* sort the threads alphabetically\n\nAuthor: CodingCat <zhunansjtu@gmail.com>\n\nCloses #7808 from CodingCat/SPARK-8416 and squashes the following commits:\n\n34fc708 [CodingCat] fix className\nd7b79dd [CodingCat] lowercase threadName\nd032882 [CodingCat] sort alphabetically and change the css class name\nf0513b1 [CodingCat] change the color & group threads by name\n2da6e06 [CodingCat] small fix\n3fc9f36 [CodingCat] define classes in webui.css\n8ee125e [CodingCat] highlight and put on top the executor threads in thread dumping page","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/3b0e44490aebfba30afc147e4a34a63439d985c6"},{"sha":"5eb89f67e323dcf9fa3d5b30f9b5cb8f10ca1e8c","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9577][SQL] Surface concrete iterator types in various sort classes.\n\nWe often return abstract iterator types in various sort-related classes (e.g. UnsafeKVExternalSorter). It is actually better to return a more concrete type, so the callsite uses that type and JIT can inline the iterator calls.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #7911 from rxin/surface-concrete-type and squashes the following commits:\n\n0422add [Reynold Xin] [SPARK-9577][SQL] Surface concrete iterator types in various sort classes.","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/5eb89f67e323dcf9fa3d5b30f9b5cb8f10ca1e8c"},{"sha":"0afa6fbf525723e97c6dacfdba3ad1762637ffa9","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-9521] [DOCS] Addendum. Require Maven 3.3.3+ in the build\n\nFollow on for #7852: Building Spark doc needs to refer to new Maven requirement too\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #7905 from srowen/SPARK-9521.2 and squashes the following commits:\n\n73285df [Sean Owen] Follow on for #7852: Building Spark doc needs to refer to new Maven requirement too","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/0afa6fbf525723e97c6dacfdba3ad1762637ffa9"},{"sha":"9e952ecbce670e9b532a1c664a4d03b66e404112","author":{"email":"f264844b57730cf91a9068607b05f0fbe2cff97a@gmail.com","name":"Ankur Dave"},"message":"[SPARK-3190] [GRAPHX] Fix VertexRDD.count() overflow regression\n\nSPARK-3190 was originally fixed by 96df92906978c5f58e0cc8ff5eebe5b35a08be3b, but a5ef58113667ff73562ce6db381cff96a0b354b0 introduced a regression during refactoring. This commit fixes the regression.\n\nAuthor: Ankur Dave <ankurdave@gmail.com>\n\nCloses #7923 from ankurdave/SPARK-3190-reopening and squashes the following commits:\n\na3e1b23 [Ankur Dave] Fix VertexRDD.count() overflow regression","distinct":false,"url":"https://api.github.com/repos/viirya/spark-1/commits/9e952ecbce670e9b532a1c664a4d03b66e404112"}]}