{"push_id":886867765,"size":6,"distinct_size":6,"ref":"refs/heads/branch-1.6","head":"c47a7373a88f49c77b8d65e887cac2ef1ae22eae","before":"1b3db967e05a628897b7162aa605b2e4650a0d58","commits":[{"sha":"72da2a21f0940b97757ace5975535e559d627688","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-8414] Ensure context cleaner periodic cleanups\n\nGarbage collection triggers cleanups. If the driver JVM is huge and there is little memory pressure, we may never clean up shuffle files on executors. This is a problem for long-running applications (e.g. streaming).\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #10070 from andrewor14/periodic-gc.\n\n(cherry picked from commit 1ce4adf55b535518c2e63917a827fac1f2df4e8e)\nSigned-off-by: Josh Rosen <joshrosen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/72da2a21f0940b97757ace5975535e559d627688"},{"sha":"84c44b500b5c90dffbe1a6b0aa86f01699b09b96","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-12081] Make unified memory manager work with small heaps\n\nThe existing `spark.memory.fraction` (default 0.75) gives the system 25% of the space to work with. For small heaps, this is not enough: e.g. default 1GB leaves only 250MB system memory. This is especially a problem in local mode, where the driver and executor are crammed in the same JVM. Members of the community have reported driver OOM's in such cases.\n\n**New proposal.** We now reserve 300MB before taking the 75%. For 1GB JVMs, this leaves `(1024 - 300) * 0.75 = 543MB` for execution and storage. This is proposal (1) listed in the [JIRA](https://issues.apache.org/jira/browse/SPARK-12081).\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #10081 from andrewor14/unified-memory-small-heaps.\n\n(cherry picked from commit d96f8c997b9bb5c3d61f513d2c71d67ccf8e85d6)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/84c44b500b5c90dffbe1a6b0aa86f01699b09b96"},{"sha":"a5743affcf73f7bf71517171583cbddc44cc9368","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12077][SQL] change the default plan for single distinct\n\nUse try to match the behavior for single distinct aggregation with Spark 1.5, but that's not scalable, we should be robust by default, have a flag to address performance regression for low cardinality aggregation.\n\ncc yhuai nongli\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10075 from davies/agg_15.\n\n(cherry picked from commit 96691feae0229fd693c29475620be2c4059dd080)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/a5743affcf73f7bf71517171583cbddc44cc9368"},{"sha":"1f42295b5df69a6039ed2ba8ea67a8e57d77644d","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-12087][STREAMING] Create new JobConf for every batch in saveAsHadoopFiles\n\nThe JobConf object created in `DStream.saveAsHadoopFiles` is used concurrently in multiple places:\n* The JobConf is updated by `RDD.saveAsHadoopFile()` before the job is launched\n* The JobConf is serialized as part of the DStream checkpoints.\nThese concurrent accesses (updating in one thread, while the another thread is serializing it) can lead to concurrentModidicationException in the underlying Java hashmap using in the internal Hadoop Configuration object.\n\nThe solution is to create a new JobConf in every batch, that is updated by `RDD.saveAsHadoopFile()`, while the checkpointing serializes the original JobConf.\n\nTests to be added in #9988 will fail reliably without this patch. Keeping this patch really small to make sure that it can be added to previous branches.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #10088 from tdas/SPARK-12087.\n\n(cherry picked from commit 8a75a3049539eeef04c0db51736e97070c162b46)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/1f42295b5df69a6039ed2ba8ea67a8e57d77644d"},{"sha":"3c4938e26185dc0637f3af624830dbff11589997","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-11949][SQL] Check bitmasks to set nullable property\n\nFollowing up #10038.\n\nWe can use bitmasks to determine which grouping expressions need to be set as nullable.\n\ncc yhuai\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #10067 from viirya/fix-cube-following.\n\n(cherry picked from commit 0f37d1d7ed7f6e34f98f2a3c274918de29e7a1d7)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/3c4938e26185dc0637f3af624830dbff11589997"},{"sha":"c47a7373a88f49c77b8d65e887cac2ef1ae22eae","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12090] [PYSPARK] consider shuffle in coalesce()\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10090 from davies/fix_coalesce.\n\n(cherry picked from commit 4375eb3f48fc7ae90caf6c21a0d3ab0b66bf4efa)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/c47a7373a88f49c77b8d65e887cac2ef1ae22eae"}]}