{"push_id":817731444,"size":64,"distinct_size":24,"ref":"refs/heads/SPARK-9841-params-clear-needs-to-be-public","head":"1447dd5d8d716e1a20a6f2a3c56378773461158d","before":"6866a2affac26f251fd5b20a037b963316040150","commits":[{"sha":"067afb4e9bb227f159bcbc2aafafce9693303ea9","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-10699] [ML] Support checkpointInterval can be disabled\n\nCurrently use can set ```checkpointInterval``` to specify how often should the cache be check-pointed. But we also need the function that users can disable it. This PR supports that users can disable checkpoint if user setting ```checkpointInterval = -1```.\nWe also add documents for GBT ```cacheNodeIds``` to make users can understand more clearly about checkpoint.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8820 from yanboliang/spark-10699.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/067afb4e9bb227f159bcbc2aafafce9693303ea9"},{"sha":"9952217749118ae78fe794ca11e1c4a87a4ae8ba","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-10731] [SQL] Delegate to Scala's DataFrame.take implementation in Python DataFrame.\n\nPython DataFrame.head/take now requires scanning all the partitions. This pull request changes them to delegate the actual implementation to Scala DataFrame (by calling DataFrame.take).\n\nThis is more of a hack for fixing this issue in 1.5.1. A more proper fix is to change executeCollect and executeTake to return InternalRow rather than Row, and thus eliminate the extra round-trip conversion.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #8876 from rxin/SPARK-10731.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/9952217749118ae78fe794ca11e1c4a87a4ae8ba"},{"sha":"084e4e126211d74a79e8dbd2d0e604dd3c650822","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-6028][Core]A new RPC implemetation based on the network module\n\nDesign doc: https://docs.google.com/document/d/1CF5G6rGVQMKSyV_QKo4D2M-x6rxz5x1Ew7aK3Uq6u8c/edit?usp=sharing\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #6457 from zsxwing/new-rpc.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/084e4e126211d74a79e8dbd2d0e604dd3c650822"},{"sha":"83f6f54d12a418f5158ee7ee985b54eef8cc1cf0","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-10474] [SQL] Aggregation fails to allocate memory for pointer array (round 2)\n\nThis patch reverts most of the changes in a previous fix #8827.\n\nThe real cause of the issue is that in `TungstenAggregate`'s prepare method we only reserve 1 page, but later when we switch to sort-based aggregation we try to acquire 1 page AND a pointer array. The longer-term fix should be to reserve also the pointer array, but for now ***we will simply not track the pointer array***. (Note that elsewhere we already don't track the pointer array, e.g. [here](https://github.com/apache/spark/blob/a18208047f06a4244703c17023bb20cbe1f59d73/sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java#L88))\n\nNote: This patch reuses the unit test added in #8827 so it doesn't show up in the diff.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #8888 from andrewor14/dont-track-pointer-array.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/83f6f54d12a418f5158ee7ee985b54eef8cc1cf0"},{"sha":"758c9d25e92417f8c06328c3af7ea2ef0212c79f","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-10692] [STREAMING] Expose failureReasons in BatchInfo for streaming UI to clear failed batches\n\nSlightly modified version of #8818, all credit goes to zsxwing\n\nAuthor: zsxwing <zsxwing@gmail.com>\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #8892 from tdas/SPARK-10692.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/758c9d25e92417f8c06328c3af7ea2ef0212c79f"},{"sha":"d91967e159f416924bbd7f0db25156588d4bd7b1","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-10763] [ML] [JAVA] [TEST] Update Java MLLIB/ML tests to use simplified dataframe construction\n\nAs introduced in https://issues.apache.org/jira/browse/SPARK-10630 we now have an easier way to create dataframes from local Java lists. Lets update the tests to use those.\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #8886 from holdenk/SPARK-10763-update-java-mllib-ml-tests-to-use-simplified-dataframe-construction.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/d91967e159f416924bbd7f0db25156588d4bd7b1"},{"sha":"02144d6745ec0a6d8877d969feb82139bd22437f","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"Revert \"[SPARK-6028][Core]A new RPC implemetation based on the network module\"\n\nThis reverts commit 084e4e126211d74a79e8dbd2d0e604dd3c650822.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/02144d6745ec0a6d8877d969feb82139bd22437f"},{"sha":"341b13f8f5eb118f1fb4d4f84418715ac4750a4d","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@163.com","name":"Wenchen Fan"},"message":"[SPARK-10765] [SQL] use new aggregate interface for hive UDAF\n\nAuthor: Wenchen Fan <cloud0fan@163.com>\n\nCloses #8874 from cloud-fan/hive-agg.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/341b13f8f5eb118f1fb4d4f84418715ac4750a4d"},{"sha":"b3862d3c59746ffb5f089aea4ff9e6f033a2c658","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-10705] [SQL] Avoid using external rows in DataFrame.toJSON\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-10705\n\nAs described in the JIRA ticket, `DataFrame.toJSON` uses `DataFrame.mapPartitions`, which converts internal rows to external rows. We should use `queryExecution.toRdd.mapPartitions` that directly uses internal rows for better performance.\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #8865 from viirya/df-tojson-internalrow.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/b3862d3c59746ffb5f089aea4ff9e6f033a2c658"},{"sha":"8023242e77e4a799de6edc490078285684549b6d","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-10761] Refactor DiskBlockObjectWriter to not require BlockId\n\nThe DiskBlockObjectWriter constructor took a BlockId parameter but never used it. As part of some general cleanup in these interfaces, this patch refactors its constructor to eliminate this parameter.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #8871 from JoshRosen/disk-block-object-writer-blockid-cleanup.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/8023242e77e4a799de6edc490078285684549b6d"},{"sha":"21fd12cb17b9e08a0cc49b4fda801af947a4183b","author":{"email":"c481c5e387de8107aa9fe99c851e11e1ff44d0c3@databricks.com","name":"Matei Zaharia"},"message":"[SPARK-9852] Let reduce tasks fetch multiple map output partitions\n\nThis makes two changes:\n\n- Allow reduce tasks to fetch multiple map output partitions -- this is a pretty small change to HashShuffleFetcher\n- Move shuffle locality computation out of DAGScheduler and into ShuffledRDD / MapOutputTracker; this was needed because the code in DAGScheduler wouldn't work for RDDs that fetch multiple map output partitions from each reduce task\n\nI also added an AdaptiveSchedulingSuite that creates RDDs depending on multiple map output partitions.\n\nAuthor: Matei Zaharia <matei@databricks.com>\n\nCloses #8844 from mateiz/spark-9852.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/21fd12cb17b9e08a0cc49b4fda801af947a4183b"},{"sha":"922338812c03eba43f2f1a6c414d1b6b049811cf","author":{"email":"cc463b26613f4ba480f0f94ca5cb2cab678c4fcb@databricks.com","name":"Eric Liang"},"message":"[SPARK-9681] [ML] Support R feature interactions in RFormula\n\nThis integrates the Interaction feature transformer with SparkR R formula support (i.e. support `:`).\n\nTo generate reasonable ML attribute names for feature interactions, it was necessary to add the ability to read attribute the original attribute names back from `StructField`, and also to specify custom group prefixes in `VectorAssembler`. This also has the side-benefit of cleaning up the double-underscores in the attributes generated for non-interaction terms.\n\nmengxr\n\nAuthor: Eric Liang <ekl@databricks.com>\n\nCloses #8830 from ericl/interaction-2.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/922338812c03eba43f2f1a6c414d1b6b049811cf"},{"sha":"6fcee906d2afb5d5c3c49e0a669637a87e82b910","author":{"email":"24293fb0f3748cb0e5fccba5c0446bb95db892a1@gmail.com","name":"Narine Kokhlikyan"},"message":"[SPARK-10760] [SPARKR] SparkR glm: the documentation in examples - family argument is missing\n\nHi everyone,\n\nSince the family argument is required for the glm function, the execution of:\n\nmodel <- glm(Sepal_Length ~ Sepal_Width, df)\n\nis failing.\n\nI've fixed the documentation by adding the family argument and also added the summay(model) which will show the coefficients for the model.\n\nThanks,\nNarine\n\nAuthor: Narine Kokhlikyan <narine.kokhlikyan@gmail.com>\n\nCloses #8870 from NarineK/sparkrml.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/6fcee906d2afb5d5c3c49e0a669637a87e82b910"},{"sha":"6f94d56a95e8c3a410a8d0c6a24ccca043227ba9","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-10845] [SQL] Makes spark.sql.hive.version a SQLConfEntry\n\nWhen refactoring SQL options from plain strings to the strongly typed `SQLConfEntry`, `spark.sql.hive.version` wasn't migrated, and doesn't show up in the result of `SET -v`, as `SET -v` only shows public `SQLConfEntry` instances. This affects compatibility with Simba ODBC driver.\n\nThis PR migrates this SQL option as a `SQLConfEntry` to fix this issue.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #8925 from liancheng/spark-10845/hive-version-conf.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/6f94d56a95e8c3a410a8d0c6a24ccca043227ba9"},{"sha":"299b439920f980cce4c4f4e4a8436a5145efeaa3","author":{"email":"fc59c3bb020865d367e2c1c1dc47888b40f5d7a8@gmail.com","name":"y-shimizu"},"message":"[SPARK-10778] [MLLIB] Implement toString for AssociationRules.Rule\n\nI implemented toString for AssociationRules.Rule, format like `[x, y] => {z}: 1.0`\n\nAuthor: y-shimizu <y.shimizu0429@gmail.com>\n\nCloses #8904 from y-shimizu/master.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/299b439920f980cce4c4f4e4a8436a5145efeaa3"},{"sha":"418e5e4cbdaab87addb91ac0bb2245ff0213ac81","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@163.com","name":"Wenchen Fan"},"message":"[SPARK-10741] [SQL] Hive Query Having/OrderBy against Parquet table is not working\n\nhttps://issues.apache.org/jira/browse/SPARK-10741\nI choose the second approach: do not change output exprIds when convert MetastoreRelation to LogicalRelation\n\nAuthor: Wenchen Fan <cloud0fan@163.com>\n\nCloses #8889 from cloud-fan/hot-bug.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/418e5e4cbdaab87addb91ac0bb2245ff0213ac81"},{"sha":"8ecba3e86e53834413da8b4299f5791545cae12e","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-10720] [SQL] [JAVA] Add a java wrapper to create a dataframe from a local list of java beans\n\nSimilar to SPARK-10630 it would be nice if Java users didn't have to parallelize there data explicitly (as Scala users already can skip). Issue came up in http://stackoverflow.com/questions/32613413/apache-spark-machine-learning-cant-get-estimator-example-to-work\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #8879 from holdenk/SPARK-10720-add-a-java-wrapper-to-create-a-dataframe-from-a-local-list-of-java-beans.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/8ecba3e86e53834413da8b4299f5791545cae12e"},{"sha":"fb4c7be747a35ea733df6010194cd76db449adc9","author":{"email":"39a349d1b1e53fa5f58daf5bd84a74a2a895db77@gmail.com","name":"Bin Wang"},"message":"add doc for spark.streaming.stopGracefullyOnShutdown\n\nAuthor: Bin Wang <wbin00@gmail.com>\n\nCloses #8898 from wb14123/doc.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/fb4c7be747a35ea733df6010194cd76db449adc9"},{"sha":"b58249930d58e2de238c05aaf5fa9315b4c3cbab","author":{"email":"e94b1b5f03b99124a5b96c61906264666b9b6302@users.noreply.github.com","name":"David Martin"},"message":"Fix two mistakes in programming-guide page\n\nseperate -> separate\nsees -> see\n\nAuthor: David Martin <dmartinpro@users.noreply.github.com>\n\nCloses #8928 from dmartinpro/patch-1.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/b58249930d58e2de238c05aaf5fa9315b4c3cbab"},{"sha":"d8d50ed388d2e695b69d2b93a620045ef2f0bc18","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-10812] [YARN] Spark hadoop util support switching to yarn\n\nWhile this is likely not a huge issue for real production systems, for test systems which may setup a Spark Context and tear it down and stand up a Spark Context with a different master (e.g. some local mode & some yarn mode) tests this cane be an issue. Discovered during work on spark-testing-base on Spark 1.4.1, but seems like the logic that triggers it is present in master (see SparkHadoopUtil object). A valid work around for users encountering this issue is to fork a different JVM, however this can be heavy weight.\n\n```\n[info] SampleMiniClusterTest:\n[info] Exception encountered when attempting to run a suite with class name: com.holdenkarau.spark.testing.SampleMiniClusterTest *** ABORTED ***\n[info] java.lang.ClassCastException: org.apache.spark.deploy.SparkHadoopUtil cannot be cast to org.apache.spark.deploy.yarn.YarnSparkHadoopUtil\n[info] at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$.get(YarnSparkHadoopUtil.scala:163)\n[info] at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:257)\n[info] at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:561)\n[info] at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:115)\n[info] at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)\n[info] at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)\n[info] at org.apache.spark.SparkContext.<init>(SparkContext.scala:497)\n[info] at com.holdenkarau.spark.testing.SharedMiniCluster$class.setup(SharedMiniCluster.scala:186)\n[info] at com.holdenkarau.spark.testing.SampleMiniClusterTest.setup(SampleMiniClusterTest.scala:26)\n[info] at com.holdenkarau.spark.testing.SharedMiniCluster$class.beforeAll(SharedMiniCluster.scala:103)\n```\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #8911 from holdenk/SPARK-10812-spark-hadoop-util-support-switching-to-yarn.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/d8d50ed388d2e695b69d2b93a620045ef2f0bc18"}]}