{"push_id":673165431,"size":25,"distinct_size":25,"ref":"refs/heads/master","head":"039093965b10cfbd028359708cb0e88bd2bc1318","before":"258fbce3cb47c420da500ca395afd7193ce33b70","commits":[{"sha":"509d55ab416359fab0525189458e2ea96379cf14","author":{"email":"89674bb0dc6f2e03b27640b006f61a9dd0e29d81@hw11853.local","name":"Ram Sriharsha"},"message":"[SPARK-7574] [ML] [DOC] User guide for OneVsRest\n\nIncluding Iris Dataset (after shuffling and relabeling 3 -> 0 to confirm to 0 -> numClasses-1 labeling). Could not find an existing dataset in data/mllib for multiclass classification.\n\nAuthor: Ram Sriharsha <rsriharsha@hw11853.local>\n\nCloses #6296 from harsha2010/SPARK-7574 and squashes the following commits:\n\n645427c [Ram Sriharsha] cleanup\n46c41b1 [Ram Sriharsha] cleanup\n2f76295 [Ram Sriharsha] Code Review Fixes\nebdf103 [Ram Sriharsha] Java Example\nc026613 [Ram Sriharsha] Code Review fixes\n4b7d1a6 [Ram Sriharsha] minor cleanup\n13bed9c [Ram Sriharsha] add wikipedia link\nbb9dbfa [Ram Sriharsha] Clean up naming\n6f90db1 [Ram Sriharsha] [SPARK-7574][ml][doc] User guide for OneVsRest","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/509d55ab416359fab0525189458e2ea96379cf14"},{"sha":"eac00691da93a94e6cff5ae0f8952e5724e78094","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-7766] KryoSerializerInstance reuse is unsafe when auto-reset is disabled\n\nSPARK-3386 / #5606 modified the shuffle write path to re-use serializer instances across multiple calls to DiskBlockObjectWriter. It turns out that this introduced a very rare bug when using `KryoSerializer`: if auto-reset is disabled and reference-tracking is enabled, then we'll end up re-using the same serializer instance to write multiple output streams without calling `reset()` between write calls, which can lead to cases where objects in one file may contain references to objects that are in previous files, causing errors during deserialization.\n\nThis patch fixes this bug by calling `reset()` at the start of `serialize()` and `serializeStream()`. I also added a regression test which demonstrates that this problem only occurs when auto-reset is disabled and reference-tracking is enabled.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #6293 from JoshRosen/kryo-instance-reuse-bug and squashes the following commits:\n\ne19726d [Josh Rosen] Add fix for SPARK-7766.\n71845e3 [Josh Rosen] Add failing regression test to trigger Kryo re-use bug","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/eac00691da93a94e6cff5ae0f8952e5724e78094"},{"sha":"31d5d463e76b6611c854c6cf27059fec8198adc9","author":{"email":"7ea8d8bc92d239699affc4b9ea1759f960ddbaa7@huawei.com","name":"WangTaoTheTonic"},"message":"[SPARK-7758] [SQL] Override more configs to avoid failure when connect to a postgre sql\n\nhttps://issues.apache.org/jira/browse/SPARK-7758\n\nWhen initializing `executionHive`, we only masks\n`javax.jdo.option.ConnectionURL` to override metastore location.  However,\nother properties that relates to the actual Hive metastore data source are not\nmasked.  For example, when using Spark SQL with a PostgreSQL backed Hive\nmetastore, `executionHive` actually tries to use settings read from\n`hive-site.xml`, which talks about PostgreSQL, to connect to the temporary\nDerby metastore, thus causes error.\n\nTo fix this, we need to mask all metastore data source properties.\nSpecifically, according to the code of [Hive `ObjectStore.getDataSourceProps()`\nmethod] [1], all properties whose name mentions \"jdo\" and \"datanucleus\" must be\nincluded.\n\n[1]: https://github.com/apache/hive/blob/release-0.13.1/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L288\n\nHave tested using postgre sql as metastore, it worked fine.\n\nAuthor: WangTaoTheTonic <wangtao111@huawei.com>\n\nCloses #6314 from WangTaoTheTonic/SPARK-7758 and squashes the following commits:\n\nca7ae7c [WangTaoTheTonic] add comments\n86caf2c [WangTaoTheTonic] delete unused import\ne4f0feb [WangTaoTheTonic] block more data source related property\n92a81fa [WangTaoTheTonic] fix style check\ne3e683d [WangTaoTheTonic] override more configs to avoid failuer connecting to postgre sql","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/31d5d463e76b6611c854c6cf27059fec8198adc9"},{"sha":"e4aef91fe70d6c9765d530b913a9d79103fc27ce","author":{"email":"8da22c45572ff7532871766932ae66e63da3e927@mola.io","name":"Santiago M. Mola"},"message":"[SPARK-7724] [SQL] Support Intersect/Except in Catalyst DSL.\n\nAuthor: Santiago M. Mola <santi@mola.io>\n\nCloses #6327 from smola/feature/catalyst-dsl-set-ops and squashes the following commits:\n\n11db778 [Santiago M. Mola] [SPARK-7724] [SQL] Support Intersect/Except in Catalyst DSL.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/e4aef91fe70d6c9765d530b913a9d79103fc27ce"},{"sha":"126d7235de649ea5619dee6ad3a70970ee90df93","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-7270] [SQL] Consider dynamic partition when inserting into hive table\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-7270\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #5864 from viirya/dyn_partition_insert and squashes the following commits:\n\nb5627df [Liang-Chi Hsieh] For comments.\n3b21e4b [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into dyn_partition_insert\n8a4352d [Liang-Chi Hsieh] Consider dynamic partition when inserting into hive table.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/126d7235de649ea5619dee6ad3a70970ee90df93"},{"sha":"821254fb945c3e19540eb57fff1f656737ef484b","author":{"email":"b54fbb0e17aa598fdc04bb3d79ba76b5f0737666@cloudera.com","name":"Imran Rashid"},"message":"[SPARK-7760] add /json back into master & worker pages; add test\n\nAuthor: Imran Rashid <irashid@cloudera.com>\n\nCloses #6284 from squito/SPARK-7760 and squashes the following commits:\n\n5e02d8a [Imran Rashid] style; increase timeout\n9987399 [Imran Rashid] comment\n8c7ed63 [Imran Rashid] add /json back into master & worker pages; add test","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/821254fb945c3e19540eb57fff1f656737ef484b"},{"sha":"3c1305107a2d6d2de862e8b41dbad0e85585b1ef","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-7834] [SQL] Better window error messages\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #6363 from marmbrus/windowErrors and squashes the following commits:\n\n516b02d [Michael Armbrust] [SPARK-7834] [SQL] Better window error messages","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/3c1305107a2d6d2de862e8b41dbad0e85585b1ef"},{"sha":"3d8760d76eae41dcaab8e9aeda19619f3d5f1596","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-7771] [SPARK-7779] Dynamic allocation: lower default timeouts further\n\nThe default add time of 5s is still too slow for small jobs. Also, the current default remove time of 10 minutes seem rather high. This patch lowers both and rephrases a few log messages.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #6301 from andrewor14/da-minor and squashes the following commits:\n\n6d614a6 [Andrew Or] Lower log level\n2811492 [Andrew Or] Log information when requests are canceled\n5fcd3eb [Andrew Or] Fix tests\n3320710 [Andrew Or] Lower timeouts + rephrase a few log messages","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/3d8760d76eae41dcaab8e9aeda19619f3d5f1596"},{"sha":"1c388a9985999e043fa002618a357bc8f0a8b65a","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-7788] Made KinesisReceiver.onStart() non-blocking\n\nKinesisReceiver calls worker.run() which is a blocking call (while loop) as per source code of kinesis-client library - https://github.com/awslabs/amazon-kinesis-client/blob/v1.2.1/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/Worker.java.\nThis results in infinite loop while calling sparkStreamingContext.stop(stopSparkContext = false, stopGracefully = true) perhaps because ReceiverTracker is never able to register the receiver (it's receiverInfo field is a empty map) causing it to be stuck in infinite loop while waiting for running flag to be set to false.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #6348 from tdas/SPARK-7788 and squashes the following commits:\n\n2584683 [Tathagata Das] Added receiver id in thread name\n6cf1cd4 [Tathagata Das] Made KinesisReceiver.onStart non-blocking","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/1c388a9985999e043fa002618a357bc8f0a8b65a"},{"sha":"8014e1f6bb871d9fd4db74106eb4425d0c1e9dd6","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":"[SPARK-7224] [SPARK-7306] mock repository generator for --packages tests without nio.Path\n\nThe previous PR for SPARK-7224 (#5790) broke JDK 6, because it used java.nio.Path, which was in jdk 7, and not in 6. This PR uses Guava's `Files` to handle directory creation, and etc...\n\nThe description from the previous PR:\n> This patch contains an `IvyTestUtils` file, which dynamically generates jars and pom files to test the `--packages` feature without having to rely on the internet, and Maven Central.\n\ncc pwendell\n\nI also rand the flaky test about 20 times locally, it didn't fail a single time, but I think it may fail like once every 100 builds? I still haven't figured the cause yet, but the test before it, `--jars` was also failing after we turned off the `--packages` test in `SparkSubmitSuite`. It may be related to the launch of SparkSubmit.\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #5892 from brkyvz/maven-utils and squashes the following commits:\n\ne9b1903 [Burak Yavuz] fix merge conflict\n68214e0 [Burak Yavuz] remove ignore for test(neglect spark dependencies)\ne632381 [Burak Yavuz] fix ignore\n9ef1408 [Burak Yavuz] re-enable --packages test\n22eea62 [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into maven-utils\n05cd0de [Burak Yavuz] added mock repository generator","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/8014e1f6bb871d9fd4db74106eb4425d0c1e9dd6"},{"sha":"63a5ce75eac48a297751ac505d70ce4d47daf903","author":{"email":"fcf9e35ad50aab44387f6ff709c4c4bf73ee4abe@gmail.com","name":"Mike Dusenberry"},"message":"[SPARK-7830] [DOCS] [MLLIB] Adding logistic regression to the list of Multiclass Classification Supported Methods documentation\n\nAdded logistic regression to the list of Multiclass Classification Supported Methods in the MLlib Classification and Regression documentation, as it was missing.\n\nAuthor: Mike Dusenberry <dusenberrymw@gmail.com>\n\nCloses #6357 from dusenberrymw/Add_LR_To_List_Of_Multiclass_Classification_Methods and squashes the following commits:\n\n7918650 [Mike Dusenberry] Updating broken link due to the \"Binary Classification\" section on the Linear Methods page being renamed to \"Classification\".\n3005dc2 [Mike Dusenberry] Adding logistic regression to the list of Multiclass Classification Supported Methods in the MLlib Classification and Regression documentation, as it was missing.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/63a5ce75eac48a297751ac505d70ce4d47daf903"},{"sha":"a16357413d2823bcc1d1bf55b4da191dc9b1b69a","author":{"email":"e555c05d4d1bb98dca0cd879ab4e297beca703a7@quantcast.com","name":"Akshat Aranya"},"message":"[SPARK-7795] [CORE] Speed up task scheduling in standalone mode by reusing serializer\n\nMy experiments with scheduling very short tasks in standalone cluster mode indicated that a significant amount of time was being spent in scheduling the tasks (>500ms for 256 tasks).  I found that most of the time was being spent in creating a new instance of serializer for each task.  Changing this to just one serializer brought down the scheduling time to 8ms.\n\nAuthor: Akshat Aranya <aaranya@quantcast.com>\n\nCloses #6323 from coolfrood/master and squashes the following commits:\n\n12d8c9e [Akshat Aranya] Reduce visibility of serializer\nbd4a5dd [Akshat Aranya] Style fix\n0b8ca93 [Akshat Aranya] Incorporate review comments\nfe530cd [Akshat Aranya] Speed up task scheduling in standalone mode by reusing serializer instead of creating a new one for each task.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/a16357413d2823bcc1d1bf55b4da191dc9b1b69a"},{"sha":"017b3404a50bd4b04ed73c5a69acb7b19a929822","author":{"email":"ad63b47dfac1951bde2b661ba9fbbe123f686bee@cs.berkeley.edu","name":"Shivaram Venkataraman"},"message":"[MINOR] Add SparkR to create-release script\n\nEnables the SparkR profiles for all the binary builds we create\n\ncc pwendell\n\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\n\nCloses #6371 from shivaram/sparkr-create-release and squashes the following commits:\n\nca5a0b2 [Shivaram Venkataraman] Add -Psparkr to create-release.sh","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/017b3404a50bd4b04ed73c5a69acb7b19a929822"},{"sha":"baa89838cca96fa091c9e5ce62be01e1a265d820","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-7838] [STREAMING] Set scope for kinesis stream\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #6369 from tdas/SPARK-7838 and squashes the following commits:\n\n87d1c7f [Tathagata Das] Addressed comment\n37775d8 [Tathagata Das] set scope for kinesis stream","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/baa89838cca96fa091c9e5ce62be01e1a265d820"},{"sha":"368b8c2b5ed8b06b00ac87059f75915b13ba3b8d","author":{"email":"e1f8d9790f67825b23faf6a373ab7c83bc8900f2@apache.org","name":"Hari Shreedharan"},"message":"[HOTFIX] Add tests for SparkListenerApplicationStart with Driver Logs.\n\n#6166 added the driver logs to `SparkListenerApplicationStart`. This  adds tests in `JsonProtocolSuite` to ensure we don't regress.\n\nAuthor: Hari Shreedharan <hshreedharan@apache.org>\n\nCloses #6368 from harishreedharan/jsonprotocol-test and squashes the following commits:\n\ndc9eafc [Hari Shreedharan] [HOTFIX] Add tests for SparkListenerApplicationStart with Driver Logs.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/368b8c2b5ed8b06b00ac87059f75915b13ba3b8d"},{"sha":"4583cf4be17155c68178155acf6866d7cc8f7df0","author":{"email":"2f805403620fd0413e09d4227afe73325dc01d5f@gmail.com","name":"GenTang"},"message":"[SPARK-5090] [EXAMPLES] The improvement of python converter for hbase\n\nHi,\n\nFollowing the discussion in http://apache-spark-developers-list.1001551.n3.nabble.com/python-converter-in-HBaseConverter-scala-spark-examples-td10001.html. I made some modification in three files in package examples:\n1. HBaseConverters.scala: the new converter will converts all the records in an hbase results into a single string\n2. hbase_input.py: as the value string may contain several records, we can use ast package to convert the string into dict\n3. HBaseTest.scala: as the package examples use hbase 0.98.7 the original constructor HTableDescriptor is deprecated. The updation to new constructor is made\n\nAuthor: GenTang <gen.tang86@gmail.com>\n\nCloses #3920 from GenTang/master and squashes the following commits:\n\nd2153df [GenTang] import JSONObject precisely\n4802481 [GenTang] dump the result into a singl String\n62df7f0 [GenTang] remove the comment\n21de653 [GenTang] return the string in json format\n15b1fe3 [GenTang] the modification of comments\n5cbbcfc [GenTang] the improvement of pythonconverter\nceb31c5 [GenTang] the modification for adapting updation of hbase\n3253b61 [GenTang] the modification accompanying the improvement of pythonconverter","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/4583cf4be17155c68178155acf6866d7cc8f7df0"},{"sha":"7af3818c6b2bf35bfa531ab7cc3a4a714385015e","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-6806] [SPARKR] [DOCS] Fill in SparkR examples in programming guide\n\nsqlCtx -> sqlContext\n\nYou can check the docs by:\n\n```\n$ cd docs\n$ SKIP_SCALADOC=1 jekyll serve\n```\ncc shivaram\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #5442 from davies/r_docs and squashes the following commits:\n\n7a12ec6 [Davies Liu] remove rdd in R docs\n8496b26 [Davies Liu] remove the docs related to RDD\ne23b9d6 [Davies Liu] delete R docs for RDD API\n222e4ff [Davies Liu] Merge branch 'master' into r_docs\n89684ce [Davies Liu] Merge branch 'r_docs' of github.com:davies/spark into r_docs\nf0a10e1 [Davies Liu] address comments from @shivaram\nf61de71 [Davies Liu] Update pairRDD.R\n3ef7cf3 [Davies Liu] use + instead of function(a,b) a+b\n2f10a77 [Davies Liu] address comments from @cafreeman\n9c2a062 [Davies Liu] mention R api together with Python API\n23f751a [Davies Liu] Fill in SparkR examples in programming guide","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/7af3818c6b2bf35bfa531ab7cc3a4a714385015e"},{"sha":"a40bca0111de45763c3ef4270afb2185c16b8f95","author":{"email":"ad63b47dfac1951bde2b661ba9fbbe123f686bee@cs.berkeley.edu","name":"Shivaram Venkataraman"},"message":"[SPARK-6811] Copy SparkR lib in make-distribution.sh\n\nThis change also remove native libraries from SparkR to make sure our distribution works across platforms\n\nTested by building on Mac, running on Amazon Linux (CentOS), Windows VM and vice-versa (built on Linux run on Mac)\n\nI will also test this with YARN soon and update this PR.\n\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\n\nCloses #6373 from shivaram/sparkr-binary and squashes the following commits:\n\nae41b5c [Shivaram Venkataraman] Remove native libraries from SparkR Also include the built SparkR package in make-distribution.sh","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/a40bca0111de45763c3ef4270afb2185c16b8f95"},{"sha":"ad0badba1450295982738934da2cc121cde18213","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-7777][Streaming] Handle the case when there is no block in a batch\n\nIn the old implementation, if a batch has no block, `areWALRecordHandlesPresent` will be `true` and it will return `WriteAheadLogBackedBlockRDD`.\n\nThis PR handles this case by returning `WriteAheadLogBackedBlockRDD` or `BlockRDD` according to the configuration.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #6372 from zsxwing/SPARK-7777 and squashes the following commits:\n\n788f895 [zsxwing] Handle the case when there is no block in a batch","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/ad0badba1450295982738934da2cc121cde18213"},{"sha":"efe3bfdf496aa6206ace2697e31dd4c0c3c824fb","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-7322, SPARK-7836, SPARK-7822][SQL] DataFrame window function related updates\n\n1. ntile should take an integer as parameter.\n2. Added Python API (based on #6364)\n3. Update documentation of various DataFrame Python functions.\n\nAuthor: Davies Liu <davies@databricks.com>\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #6374 from rxin/window-final and squashes the following commits:\n\n69004c7 [Reynold Xin] Style fix.\n288cea9 [Reynold Xin] Update documentaiton.\n7cb8985 [Reynold Xin] Merge pull request #6364 from davies/window\n66092b4 [Davies Liu] update docs\ned73cb4 [Reynold Xin] [SPARK-7322][SQL] Improve DataFrame window function documentation.\nef55132 [Davies Liu] Merge branch 'master' of github.com:apache/spark into window4\n8936ade [Davies Liu] fix maxint in python 3\n2649358 [Davies Liu] update docs\n778e2c0 [Davies Liu] SPARK-7836 and SPARK-7822: Python API of window functions","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/efe3bfdf496aa6206ace2697e31dd4c0c3c824fb"}]}