{"push_id":856649240,"size":18,"distinct_size":18,"ref":"refs/heads/master","head":"7f741905b06ed6d3dfbff6db41a3355dab71aa3c","before":"bc5d6c03893a9bd340d6b94d3550e25648412241","commits":[{"sha":"253e87e8ab8717ffef40a6d0d376b1add155ef90","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11453][SQL][FOLLOW-UP] remove DecimalLit\n\nA cleanup for https://github.com/apache/spark/pull/9085.\n\nThe `DecimalLit` is very similar to `FloatLit`, we can just keep one of them.\nAlso added low level unit test at `SqlParserSuite`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9482 from cloud-fan/parser.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/253e87e8ab8717ffef40a6d0d376b1add155ef90"},{"sha":"cf69ce136590fea51843bc54f44f0f45c7d0ac36","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-11511][STREAMING] Fix NPE when an InputDStream is not used\n\nJust ignored `InputDStream`s that have null `rememberDuration` in `DStreamGraph.getMaxInputStreamRememberDuration`.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #9476 from zsxwing/SPARK-11511.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/cf69ce136590fea51843bc54f44f0f45c7d0ac36"},{"sha":"574141a29835ce78d68c97bb54336cf4fd3c39d3","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-9162] [SQL] Implement code generation for ScalaUDF\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-9162\n\nCurrently ScalaUDF extends CodegenFallback and doesn't provide code generation implementation. This path implements code generation for ScalaUDF.\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #9270 from viirya/scalaudf-codegen.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/574141a29835ce78d68c97bb54336cf4fd3c39d3"},{"sha":"c048929c6a9f7ce57f384037cd6c0bf5751c447a","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-10978][SQL][FOLLOW-UP] More comprehensive tests for PR #9399\n\nThis PR adds test cases that test various column pruning and filter push-down cases.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #9468 from liancheng/spark-10978.follow-up.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/c048929c6a9f7ce57f384037cd6c0bf5751c447a"},{"sha":"8211aab0793cf64202b99be4f31bb8a9ae77050d","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-9858][SQL] Add an ExchangeCoordinator to estimate the number of post-shuffle partitions for aggregates and joins (follow-up)\n\nhttps://issues.apache.org/jira/browse/SPARK-9858\n\nThis PR is the follow-up work of https://github.com/apache/spark/pull/9276. It addresses JoshRosen's comments.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #9453 from yhuai/numReducer-followUp.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/8211aab0793cf64202b99be4f31bb8a9ae77050d"},{"sha":"62bb290773c9f9fa53cbe6d4eedc6e153761a763","author":{"email":"5cafd55dd0ca8119e8e1d62a6779a4b9bbdd477d@deepsense.io","name":"Jacek Laskowski"},"message":"Typo fixes + code readability improvements\n\nAuthor: Jacek Laskowski <jacek.laskowski@deepsense.io>\n\nCloses #9501 from jaceklaskowski/typos-with-style.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/62bb290773c9f9fa53cbe6d4eedc6e153761a763"},{"sha":"49f1a820372d1cba41f3f00d07eb5728f2ed6705","author":{"email":"b54fbb0e17aa598fdc04bb3d79ba76b5f0737666@cloudera.com","name":"Imran Rashid"},"message":"[SPARK-10116][CORE] XORShiftRandom.hashSeed is random in high bits\n\nhttps://issues.apache.org/jira/browse/SPARK-10116\n\nThis is really trivial, just happened to notice it -- if `XORShiftRandom.hashSeed` is really supposed to have random bits throughout (as the comment implies), it needs to do something for the conversion to `long`.\n\nmengxr mkolod\n\nAuthor: Imran Rashid <irashid@cloudera.com>\n\nCloses #8314 from squito/SPARK-10116.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/49f1a820372d1cba41f3f00d07eb5728f2ed6705"},{"sha":"f328fedafd7bd084470a5e402de0429b5b7f8cd7","author":{"email":"679c4ded150c6b9a8052807f108ce836961fed8d@questtec.nl","name":"Herman van Hovell"},"message":"[SPARK-11450] [SQL] Add Unsafe Row processing to Expand\n\nThis PR enables the Expand operator to process and produce Unsafe Rows.\n\nAuthor: Herman van Hovell <hvanhovell@questtec.nl>\n\nCloses #9414 from hvanhovell/SPARK-11450.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/f328fedafd7bd084470a5e402de0429b5b7f8cd7"},{"sha":"3a652f691b220fada0286f8d0a562c5657973d4d","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11561][SQL] Rename text data source's column name to value.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9527 from rxin/SPARK-11561.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/3a652f691b220fada0286f8d0a562c5657973d4d"},{"sha":"c447c9d54603890db7399fb80adc9fae40b71f64","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-11217][ML] save/load for non-meta estimators and transformers\n\nThis PR implements the default save/load for non-meta estimators and transformers using the JSON serialization of param values. The saved metadata includes:\n\n* class name\n* uid\n* timestamp\n* paramMap\n\nThe save/load interface is similar to DataFrames. We use the current active context by default, which should be sufficient for most use cases.\n\n~~~scala\ninstance.save(\"path\")\ninstance.write.context(sqlContext).overwrite().save(\"path\")\n\nInstance.load(\"path\")\n~~~\n\nThe param handling is different from the design doc. We didn't save default and user-set params separately, and when we load it back, all parameters are user-set. This does cause issues. But it also cause other issues if we modify the default params.\n\nTODOs:\n\n* [x] Java test\n* [ ] a follow-up PR to implement default save/load for all non-meta estimators and transformers\n\ncc jkbradley\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #9454 from mengxr/SPARK-11217.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/c447c9d54603890db7399fb80adc9fae40b71f64"},{"sha":"f6680cdc5d2912dea9768ef5c3e2cc101b06daf8","author":{"email":"a1a405b82800b492a7268cba6bc58270d2b527de@staydecay.corp.gq1.yahoo.com","name":"Thomas Graves"},"message":"[SPARK-11555] spark on yarn spark-class --num-workers doesn't work\n\nI tested the various options with both spark-submit and spark-class of specifying number of executors in both client and cluster mode where it applied.\n\n--num-workers, --num-executors, spark.executor.instances, SPARK_EXECUTOR_INSTANCES, default nothing supplied\n\nAuthor: Thomas Graves <tgraves@staydecay.corp.gq1.yahoo.com>\n\nCloses #9523 from tgravescs/SPARK-11555.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/f6680cdc5d2912dea9768ef5c3e2cc101b06daf8"},{"sha":"7e9a9e603abce8689938bdd62d04b29299644aa4","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11269][SQL] Java API support & test cases for Dataset\n\nThis simply brings https://github.com/apache/spark/pull/9358 up-to-date.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9528 from rxin/dataset-java.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/7e9a9e603abce8689938bdd62d04b29299644aa4"},{"sha":"1ab72b08601a1c8a674bdd3fab84d9804899b2c7","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@databricks.com","name":"Nong Li"},"message":"[SPARK-11410] [PYSPARK] Add python bindings for repartition and sortW…\n\n…ithinPartitions.\n\nAuthor: Nong Li <nong@databricks.com>\n\nCloses #9504 from nongli/spark-11410.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/1ab72b08601a1c8a674bdd3fab84d9804899b2c7"},{"sha":"6d0ead322e72303c6444c6ac641378a4690cde96","author":{"email":"679c4ded150c6b9a8052807f108ce836961fed8d@questtec.nl","name":"Herman van Hovell"},"message":"[SPARK-9241][SQL] Supporting multiple DISTINCT columns (2) - Rewriting Rule\n\nThe second PR for SPARK-9241, this adds support for multiple distinct columns to the new aggregation code path.\n\nThis PR solves the multiple DISTINCT column problem by rewriting these Aggregates into an Expand-Aggregate-Aggregate combination. See the [JIRA ticket](https://issues.apache.org/jira/browse/SPARK-9241) for some information on this. The advantages over the - competing - [first PR](https://github.com/apache/spark/pull/9280) are:\n- This can use the faster TungstenAggregate code path.\n- It is impossible to OOM due to an ```OpenHashSet``` allocating to much memory. However, this will multiply the number of input rows by the number of distinct clauses (plus one), and puts a lot more memory pressure on the aggregation code path itself.\n\nThe location of this Rule is a bit funny, and should probably change when the old aggregation path is changed.\n\ncc yhuai - Could you also tell me where to add tests for this?\n\nAuthor: Herman van Hovell <hvanhovell@questtec.nl>\n\nCloses #9406 from hvanhovell/SPARK-9241-rewriter.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/6d0ead322e72303c6444c6ac641378a4690cde96"},{"sha":"1c80d66e52c0bcc4e5adda78b3d8e5bf55e4f128","author":{"email":"04ade2e3a4ce65c5ee0cbfa08dacc69e7d690176@apache.org","name":"navis.ryu"},"message":"[SPARK-11546] Thrift server makes too many logs about result schema\n\nSparkExecuteStatementOperation logs result schema for each getNextRowSet() calls which is by default every 1000 rows, overwhelming whole log file.\n\nAuthor: navis.ryu <navis@apache.org>\n\nCloses #9514 from navis/SPARK-11546.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/1c80d66e52c0bcc4e5adda78b3d8e5bf55e4f128"},{"sha":"105732dcc6b651b9779f4a5773a759c5b4fbd21d","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[HOTFIX] Fix python tests after #9527\n\n#9527 missed updating the python tests.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #9533 from marmbrus/hotfixTextValue.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/105732dcc6b651b9779f4a5773a759c5b4fbd21d"},{"sha":"30b706b7b36482921ec04145a0121ca147984fa8","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-11389][CORE] Add support for off-heap memory to MemoryManager\n\nIn order to lay the groundwork for proper off-heap memory support in SQL / Tungsten, we need to extend our MemoryManager to perform bookkeeping for off-heap memory.\n\n## User-facing changes\n\nThis PR introduces a new configuration, `spark.memory.offHeapSize` (name subject to change), which specifies the absolute amount of off-heap memory that Spark and Spark SQL can use. If Tungsten is configured to use off-heap execution memory for allocating data pages, then all data page allocations must fit within this size limit.\n\n## Internals changes\n\nThis PR contains a lot of internal refactoring of the MemoryManager. The key change at the heart of this patch is the introduction of a `MemoryPool` class (name subject to change) to manage the bookkeeping for a particular category of memory (storage, on-heap execution, and off-heap execution). These MemoryPools are not fixed-size; they can be dynamically grown and shrunk according to the MemoryManager's policies. In StaticMemoryManager, these pools have fixed sizes, proportional to the legacy `[storage|shuffle].memoryFraction`. In the new UnifiedMemoryManager, the sizes of these pools are dynamically adjusted according to its policies.\n\nThere are two subclasses of `MemoryPool`: `StorageMemoryPool` manages storage memory and `ExecutionMemoryPool` manages execution memory. The MemoryManager creates two execution pools, one for on-heap memory and one for off-heap. Instances of `ExecutionMemoryPool` manage the logic for fair sharing of their pooled memory across running tasks (in other words, the ShuffleMemoryManager-like logic has been moved out of MemoryManager and pushed into these ExecutionMemoryPool instances).\n\nI think that this design is substantially easier to understand and reason about than the previous design, where most of these responsibilities were handled by MemoryManager and its subclasses. To see this, take at look at how simple the logic in `UnifiedMemoryManager` has become: it's now very easy to see when memory is dynamically shifted between storage and execution.\n\n## TODOs\n\n- [x] Fix handful of test failures in the MemoryManagerSuites.\n- [x] Fix remaining TODO comments in code.\n- [ ] Document new configuration.\n- [x] Fix commented-out tests / asserts:\n  - [x] UnifiedMemoryManagerSuite.\n- [x] Write tests that exercise the new off-heap memory management policies.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9344 from JoshRosen/offheap-memory-accounting.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/30b706b7b36482921ec04145a0121ca147984fa8"},{"sha":"7f741905b06ed6d3dfbff6db41a3355dab71aa3c","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-11112] DAG visualization: display RDD callsite\n\n<img width=\"548\" alt=\"screen shot 2015-11-01 at 9 42 33 am\" src=\"https://cloud.githubusercontent.com/assets/2133137/10870343/2a8cd070-807d-11e5-857a-4ebcace77b5b.png\">\nmateiz sarutak\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #9398 from andrewor14/rdd-callsite.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/7f741905b06ed6d3dfbff6db41a3355dab71aa3c"}]}