{"push_id":918845965,"size":298,"distinct_size":1,"ref":"refs/heads/SPARK-12040-add-toJson-fromJson-to-Vector-Vectors-in-PySpark","head":"40de351a878b57a48de04116d5c18bef64fa7fde","before":"c9730f3e16db0e483e2496c5b1fefe84e3063bbe","commits":[{"sha":"34e7093c1131162b3aa05b65a19a633a0b5b633e","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-12065] Upgrade Tachyon from 0.8.1 to 0.8.2\n\nThis commit upgrades the Tachyon dependency from 0.8.1 to 0.8.2.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10054 from JoshRosen/upgrade-to-tachyon-0.8.2.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/34e7093c1131162b3aa05b65a19a633a0b5b633e"},{"sha":"2cef1cdfbb5393270ae83179b6a4e50c3cbf9e93","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@databricks.com","name":"Nong Li"},"message":"[SPARK-12030] Fix Platform.copyMemory to handle overlapping regions.\n\nThis bug was exposed as memory corruption in Timsort which uses copyMemory to copy\nlarge regions that can overlap. The prior implementation did not handle this case\nhalf the time and always copied forward, resulting in the data being corrupt.\n\nAuthor: Nong Li <nong@databricks.com>\n\nCloses #10068 from nongli/spark-12030.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/2cef1cdfbb5393270ae83179b6a4e50c3cbf9e93"},{"sha":"60b541ee1b97c9e5e84aa2af2ce856f316ad22b3","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-12004] Preserve the RDD partitioner through RDD checkpointing\n\nThe solution is the save the RDD partitioner in a separate file in the RDD checkpoint directory. That is, `<checkpoint dir>/_partitioner`.  In most cases, whether the RDD partitioner was recovered or not, does not affect the correctness, only reduces performance. So this solution makes a best-effort attempt to save and recover the partitioner. If either fails, the checkpointing is not affected. This makes this patch safe and backward compatible.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #9983 from tdas/SPARK-12004.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/60b541ee1b97c9e5e84aa2af2ce856f316ad22b3"},{"sha":"328b757d5d4486ea3c2e246780792d7a57ee85e5","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"Revert \"[SPARK-12060][CORE] Avoid memory copy in JavaSerializerInstance.serialize\"\n\nThis reverts commit 1401166576c7018c5f9c31e0a6703d5fb16ea339.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/328b757d5d4486ea3c2e246780792d7a57ee85e5"},{"sha":"e76431f886ae8061545b3216e8e2fb38c4db1f43","author":{"email":"7853928d03425cf3be35be2fa0c5b99641435d26@gmail.com","name":"Xusen Yin"},"message":"[SPARK-11961][DOC] Add docs of ChiSqSelector\n\nhttps://issues.apache.org/jira/browse/SPARK-11961\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\n\nCloses #9965 from yinxusen/SPARK-11961.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/e76431f886ae8061545b3216e8e2fb38c4db1f43"},{"sha":"f292018f8e57779debc04998456ec875f628133b","author":{"email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com","name":"jerryshao"},"message":"[SPARK-12002][STREAMING][PYSPARK] Fix python direct stream checkpoint recovery issue\n\nFixed a minor race condition in #10017\n\nCloses #10017\n\nAuthor: jerryshao <sshao@hortonworks.com>\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #10074 from zsxwing/review-pr10017.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/f292018f8e57779debc04998456ec875f628133b"},{"sha":"ef6790fdc3b70b9d6184ec2b3d926e4b0e4b15f6","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-12075][SQL] Speed up HiveComparisionTest by avoiding / speeding up TestHive.reset()\n\nWhen profiling HiveCompatibilitySuite, I noticed that most of the time seems to be spent in expensive `TestHive.reset()` calls. This patch speeds up suites based on HiveComparisionTest, such as HiveCompatibilitySuite, with the following changes:\n\n- Avoid `TestHive.reset()` whenever possible:\n  - Use a simple set of heuristics to guess whether we need to call `reset()` in between tests.\n  - As a safety-net, automatically re-run failed tests by calling `reset()` before the re-attempt.\n- Speed up the expensive parts of `TestHive.reset()`: loading the `src` and `srcpart` tables took roughly 600ms per test, so we now avoid this by using a simple heuristic which only loads those tables by tests that reference them. This is based on simple string matching over the test queries which errs on the side of loading in more situations than might be strictly necessary.\n\nAfter these changes, HiveCompatibilitySuite seems to run in about 10 minutes.\n\nThis PR is a revival of #6663, an earlier experimental PR from June, where I played around with several possible speedups for this suite.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10055 from JoshRosen/speculative-testhive-reset.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/ef6790fdc3b70b9d6184ec2b3d926e4b0e4b15f6"},{"sha":"47a0abc343550c855e679de12983f43e6fcc0171","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@databricks.com","name":"Nong Li"},"message":"[SPARK-11328][SQL] Improve error message when hitting this issue\n\nThe issue is that the output commiter is not idempotent and retry attempts will\nfail because the output file already exists. It is not safe to clean up the file\nas this output committer is by design not retryable. Currently, the job fails\nwith a confusing file exists error. This patch is a stop gap to tell the user\nto look at the top of the error log for the proper message.\n\nThis is difficult to test locally as Spark is hardcoded not to retry. Manually\nverified by upping the retry attempts.\n\nAuthor: Nong Li <nong@databricks.com>\nAuthor: Nong Li <nongli@gmail.com>\n\nCloses #10080 from nongli/spark-11328.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/47a0abc343550c855e679de12983f43e6fcc0171"},{"sha":"5a8b5fdd6ffa58f015cdadf3f2c6df78e0a388ad","author":{"email":"b1011540267ee7fe5a02243ad09c007ca1860809@oc0558782468.ibm.com","name":"Huaxin Gao"},"message":"[SPARK-11788][SQL] surround timestamp/date value with quotes in JDBC data source\n\nWhen query the Timestamp or Date column like the following\nval filtered = jdbcdf.where($\"TIMESTAMP_COLUMN\" >= beg && $\"TIMESTAMP_COLUMN\" < end)\nThe generated SQL query is \"TIMESTAMP_COLUMN >= 2015-01-01 00:00:00.0\"\nIt should have quote around the Timestamp/Date value such as \"TIMESTAMP_COLUMN >= '2015-01-01 00:00:00.0'\"\n\nAuthor: Huaxin Gao <huaxing@oc0558782468.ibm.com>\n\nCloses #9872 from huaxingao/spark-11788.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/5a8b5fdd6ffa58f015cdadf3f2c6df78e0a388ad"},{"sha":"5872a9d89fe2720c2bcb1fc7494136947a72581c","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-11352][SQL] Escape */ in the generated comments.\n\nhttps://issues.apache.org/jira/browse/SPARK-11352\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #10072 from yhuai/SPARK-11352.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/5872a9d89fe2720c2bcb1fc7494136947a72581c"},{"sha":"e96a70d5ab2e2b43a2df17a550fa9ed2ee0001c4","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-11596][SQL] In TreeNode's argString, if a TreeNode is not a child of the current TreeNode, we should only return the simpleString.\n\nIn TreeNode's argString, if a TreeNode is not a child of the current TreeNode, we will only return the simpleString.\n\nI tested the [following case provided by Cristian](https://issues.apache.org/jira/browse/SPARK-11596?focusedCommentId=15019241&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15019241).\n```\nval c = (1 to 20).foldLeft[Option[DataFrame]] (None) { (curr, idx) =>\n    println(s\"PROCESSING >>>>>>>>>>> $idx\")\n    val df = sqlContext.sparkContext.parallelize((0 to 10).zipWithIndex).toDF(\"A\", \"B\")\n    val union = curr.map(_.unionAll(df)).getOrElse(df)\n    union.cache()\n    Some(union)\n  }\n\nc.get.explain(true)\n```\n\nWithout the change, `c.get.explain(true)` took 100s. With the change, `c.get.explain(true)` took 26ms.\n\nhttps://issues.apache.org/jira/browse/SPARK-11596\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #10079 from yhuai/SPARK-11596.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/e96a70d5ab2e2b43a2df17a550fa9ed2ee0001c4"},{"sha":"1ce4adf55b535518c2e63917a827fac1f2df4e8e","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-8414] Ensure context cleaner periodic cleanups\n\nGarbage collection triggers cleanups. If the driver JVM is huge and there is little memory pressure, we may never clean up shuffle files on executors. This is a problem for long-running applications (e.g. streaming).\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #10070 from andrewor14/periodic-gc.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/1ce4adf55b535518c2e63917a827fac1f2df4e8e"},{"sha":"d96f8c997b9bb5c3d61f513d2c71d67ccf8e85d6","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-12081] Make unified memory manager work with small heaps\n\nThe existing `spark.memory.fraction` (default 0.75) gives the system 25% of the space to work with. For small heaps, this is not enough: e.g. default 1GB leaves only 250MB system memory. This is especially a problem in local mode, where the driver and executor are crammed in the same JVM. Members of the community have reported driver OOM's in such cases.\n\n**New proposal.** We now reserve 300MB before taking the 75%. For 1GB JVMs, this leaves `(1024 - 300) * 0.75 = 543MB` for execution and storage. This is proposal (1) listed in the [JIRA](https://issues.apache.org/jira/browse/SPARK-12081).\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #10081 from andrewor14/unified-memory-small-heaps.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/d96f8c997b9bb5c3d61f513d2c71d67ccf8e85d6"},{"sha":"96691feae0229fd693c29475620be2c4059dd080","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12077][SQL] change the default plan for single distinct\n\nUse try to match the behavior for single distinct aggregation with Spark 1.5, but that's not scalable, we should be robust by default, have a flag to address performance regression for low cardinality aggregation.\n\ncc yhuai nongli\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10075 from davies/agg_15.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/96691feae0229fd693c29475620be2c4059dd080"},{"sha":"8a75a3049539eeef04c0db51736e97070c162b46","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-12087][STREAMING] Create new JobConf for every batch in saveAsHadoopFiles\n\nThe JobConf object created in `DStream.saveAsHadoopFiles` is used concurrently in multiple places:\n* The JobConf is updated by `RDD.saveAsHadoopFile()` before the job is launched\n* The JobConf is serialized as part of the DStream checkpoints.\nThese concurrent accesses (updating in one thread, while the another thread is serializing it) can lead to concurrentModidicationException in the underlying Java hashmap using in the internal Hadoop Configuration object.\n\nThe solution is to create a new JobConf in every batch, that is updated by `RDD.saveAsHadoopFile()`, while the checkpointing serializes the original JobConf.\n\nTests to be added in #9988 will fail reliably without this patch. Keeping this patch really small to make sure that it can be added to previous branches.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #10088 from tdas/SPARK-12087.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/8a75a3049539eeef04c0db51736e97070c162b46"},{"sha":"0f37d1d7ed7f6e34f98f2a3c274918de29e7a1d7","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-11949][SQL] Check bitmasks to set nullable property\n\nFollowing up #10038.\n\nWe can use bitmasks to determine which grouping expressions need to be set as nullable.\n\ncc yhuai\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #10067 from viirya/fix-cube-following.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/0f37d1d7ed7f6e34f98f2a3c274918de29e7a1d7"},{"sha":"4375eb3f48fc7ae90caf6c21a0d3ab0b66bf4efa","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12090] [PYSPARK] consider shuffle in coalesce()\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10090 from davies/fix_coalesce.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/4375eb3f48fc7ae90caf6c21a0d3ab0b66bf4efa"},{"sha":"128c29035b4e7383cc3a9a6c7a9ab6136205ac6c","author":{"email":"86548942b2ff162d27014225aec68c3b73c8489a@surfsara.nl","name":"Jeroen Schot"},"message":"[SPARK-3580][CORE] Add Consistent Method To Get Number of RDD Partitions Across Different Languages\n\nI have tried to address all the comments in pull request https://github.com/apache/spark/pull/2447.\n\nNote that the second commit (using the new method in all internal code of all components) is quite intrusive and could be omitted.\n\nAuthor: Jeroen Schot <jeroen.schot@surfsara.nl>\n\nCloses #9767 from schot/master.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/128c29035b4e7383cc3a9a6c7a9ab6136205ac6c"},{"sha":"a1542ce2f33ad365ff437d2d3014b9de2f6670e5","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-12094][SQL] Prettier tree string for TreeNode\n\nWhen examining plans of complex queries with multiple joins, a pain point of mine is that, it's hard to immediately see the sibling node of a specific query plan node. This PR adds tree lines for the tree string of a `TreeNode`, so that the result can be visually more intuitive.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #10099 from liancheng/prettier-tree-string.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/a1542ce2f33ad365ff437d2d3014b9de2f6670e5"},{"sha":"452690ba1cc3c667bdd9f3022c43c9a10267880b","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-12001] Allow partially-stopped StreamingContext to be completely stopped\n\nIf `StreamingContext.stop()` is interrupted midway through the call, the context will be marked as stopped but certain state will have not been cleaned up. Because `state = STOPPED` will be set, subsequent `stop()` calls will be unable to finish stopping the context, preventing any new StreamingContexts from being created.\n\nThis patch addresses this issue by only marking the context as `STOPPED` once the `stop()` has successfully completed which allows `stop()` to be called a second time in order to finish stopping the context in case the original `stop()` call was interrupted.\n\nI discovered this issue by examining logs from a failed Jenkins run in which this race condition occurred in `FailureSuite`, leaking an unstoppable context and causing all subsequent tests to fail.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9982 from JoshRosen/SPARK-12001.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/452690ba1cc3c667bdd9f3022c43c9a10267880b"}]}