{"push_id":827320640,"size":18,"distinct_size":18,"ref":"refs/heads/master","head":"239c040414b95d2f8af6e7af46d4832b3f2164b2","before":"1a029cff10749c98ab60bcef31a03890417f901b","commits":[{"sha":"2b574f52d7bf51b1fe2a73086a3735b633e9083f","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-7402] [ML] JSON SerDe for standard param types\n\nThis PR implements the JSON SerDe for the following param types: `Boolean`, `Int`, `Long`, `Float`, `Double`, `String`, `Array[Int]`, `Array[Double]`, and `Array[String]`. The implementation of `Float`, `Double`, and `Array[Double]` are specialized to handle `NaN` and `Inf`s. This will be used in pipeline persistence. jkbradley\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #9090 from mengxr/SPARK-7402.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/2b574f52d7bf51b1fe2a73086a3735b633e9083f"},{"sha":"b3ffac5178795f2d8e7908b3e77e8e89f50b5f6f","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-10983] Unified memory manager\n\nThis patch unifies the memory management of the storage and execution regions such that either side can borrow memory from each other. When memory pressure arises, storage will be evicted in favor of execution. To avoid regressions in cases where storage is crucial, we dynamically allocate a fraction of space for storage that execution cannot evict. Several configurations are introduced:\n\n- **spark.memory.fraction (default 0.75)**: ​fraction of the heap space used for execution and storage. The lower this is, the more frequently spills and cached data eviction occur. The purpose of this config is to set aside memory for internal metadata, user data structures, and imprecise size estimation in the case of sparse, unusually large records.\n\n- **spark.memory.storageFraction (default 0.5)**: size of the storage region within the space set aside by `s​park.memory.fraction`. ​Cached data may only be evicted if total storage exceeds this region.\n\n- **spark.memory.useLegacyMode (default false)**: whether to use the memory management that existed in Spark 1.5 and before. This is mainly for backward compatibility.\n\nFor a detailed description of the design, see [SPARK-10000](https://issues.apache.org/jira/browse/SPARK-10000). This patch builds on top of the `MemoryManager` interface introduced in #9000.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #9084 from andrewor14/unified-memory-manager.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/b3ffac5178795f2d8e7908b3e77e8e89f50b5f6f"},{"sha":"0d1b73b78b600420121ea8e58ff659ae8b4feebe","author":{"email":"0365f6b3a191ca1f960a8b40456e71c9e5c1e128@atscale.com","name":"trystanleftwich"},"message":"[SPARK-11052] Spaces in the build dir causes failures in the build/mv…\n\n…n script\n\nAuthor: trystanleftwich <trystan@atscale.com>\n\nCloses #9065 from trystanleftwich/SPARK-11052.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/0d1b73b78b600420121ea8e58ff659ae8b4feebe"},{"sha":"ef72673b234579c161b8cbb6cafc851d9eba1bfb","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-11080] [SQL] Incorporate per-JVM id into ExprId to prevent unsafe cross-JVM comparisions\n\nIn the current implementation of named expressions' `ExprIds`, we rely on a per-JVM AtomicLong to ensure that expression ids are unique within a JVM. However, these expression ids will not be _globally_ unique. This opens the potential for id collisions if new expression ids happen to be created inside of tasks rather than on the driver.\n\nThere are currently a few cases where tasks allocate expression ids, which happen to be safe because those expressions are never compared to expressions created on the driver. In order to guard against the introduction of invalid comparisons between driver-created and executor-created expression ids, this patch extends `ExprId` to incorporate a UUID to identify the JVM that created the id, which prevents collisions.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9093 from JoshRosen/SPARK-11080.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/ef72673b234579c161b8cbb6cafc851d9eba1bfb"},{"sha":"d0482f6af33e976db237405b2a978db1b7c2fd5b","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-10932] [PROJECT INFRA] Port two minor changes to release-build.sh from scripts' old repo\n\nSpark's release packaging scripts used to live in a separate repository. Although these scripts are now part of the Spark repo, there are some minor patches made against the old repos that are missing in Spark's copy of the script. This PR ports those changes.\n\n/cc shivaram, who originally submitted these changes against https://github.com/rxin/spark-utils\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #8986 from JoshRosen/port-release-build-fixes-from-rxin-repo.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/d0482f6af33e976db237405b2a978db1b7c2fd5b"},{"sha":"3889b1c7a96da1111946fa63ad69489b83468646","author":{"email":"a8225d365811cb0449b395490c087d0ddd3c6c6f@gmail.com","name":"vectorijk"},"message":"[SPARK-11059] [ML] Change range of quantile probabilities in AFTSurvivalRegression\n\nValue of the quantile probabilities array should be in the range (0, 1) instead of [0,1]\n in `AFTSurvivalRegression.scala` according to [Discussion] (https://github.com/apache/spark/pull/8926#discussion-diff-40698242)\n\nAuthor: vectorijk <jiangkai@gmail.com>\n\nCloses #9083 from vectorijk/spark-11059.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/3889b1c7a96da1111946fa63ad69489b83468646"},{"sha":"328d1b3e4bc39cce653342e04f9e08af12dd7ed8","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-11090] [SQL] Constructor for Product types from InternalRow\n\nThis is a first draft of the ability to construct expressions that will take a catalyst internal row and construct a Product (case class or tuple) that has fields with the correct names.  Support include:\n - Nested classes\n - Maps\n - Efficiently handling of arrays of primitive types\n\nNot yet supported:\n - Case classes that require custom collection types (i.e. List instead of Seq).\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #9100 from marmbrus/productContructor.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/328d1b3e4bc39cce653342e04f9e08af12dd7ed8"},{"sha":"e170c22160bb452f98c340489ebf8390116a8cbb","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@163.com","name":"Wenchen Fan"},"message":"[SPARK-11032] [SQL] correctly handle having\n\nWe should not stop resolving having when the having condtion is resolved, or something like `count(1)` will crash.\n\nAuthor: Wenchen Fan <cloud0fan@163.com>\n\nCloses #9105 from cloud-fan/having.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/e170c22160bb452f98c340489ebf8390116a8cbb"},{"sha":"15ff85b3163acbe8052d4489a00bcf1d2332fcf0","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@163.com","name":"Wenchen Fan"},"message":"[SPARK-11068] [SQL] add callback to query execution\n\nWith this feature, we can track the query plan, time cost, exception during query execution for spark users.\n\nAuthor: Wenchen Fan <cloud0fan@163.com>\n\nCloses #9078 from cloud-fan/callback.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/15ff85b3163acbe8052d4489a00bcf1d2332fcf0"},{"sha":"ce3f9a80657751ee0bc0ed6a9b6558acbb40af4f","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-11091] [SQL] Change spark.sql.canonicalizeView to spark.sql.nativeView.\n\nhttps://issues.apache.org/jira/browse/SPARK-11091\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #9103 from yhuai/SPARK-11091.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/ce3f9a80657751ee0bc0ed6a9b6558acbb40af4f"},{"sha":"8b32885704502ab2a715cf5142d7517181074428","author":{"email":"300d085397e665a9ec270b0f82922e78a289d85d@gmail.com","name":"Monica Liu"},"message":"[SPARK-10981] [SPARKR] SparkR Join improvements\n\nI was having issues with collect() and orderBy() in Spark 1.5.0 so I used the DataFrame.R file and test_sparkSQL.R file from the Spark 1.5.1 download. I only modified the join() function in DataFrame.R to include \"full\", \"fullouter\", \"left\", \"right\", and \"leftsemi\" and added corresponding test cases in the test for join() and merge() in test_sparkSQL.R file.\nPull request because I filed this JIRA bug report:\nhttps://issues.apache.org/jira/browse/SPARK-10981\n\nAuthor: Monica Liu <liu.monica.f@gmail.com>\n\nCloses #9029 from mfliu/master.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/8b32885704502ab2a715cf5142d7517181074428"},{"sha":"390b22fad69a33eb6daee25b6b858a2e768670a5","author":{"email":"3f27eab2afd49386f0ac811afb0bc245aa92c0fa@intel.com","name":"Sun Rui"},"message":"[SPARK-10996] [SPARKR] Implement sampleBy() in DataFrameStatFunctions.\n\nAuthor: Sun Rui <rui.sun@intel.com>\n\nCloses #9023 from sun-rui/SPARK-10996.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/390b22fad69a33eb6daee25b6b858a2e768670a5"},{"sha":"135a2ce5b0b927b512c832d61c25e7b9d57e30be","author":{"email":"a1a405b82800b492a7268cba6bc58270d2b527de@yahoo-inc.com","name":"Tom Graves"},"message":"[SPARK-10619] Can't sort columns on Executor Page\n\nshould pick into spark 1.5.2 also.\n\nhttps://issues.apache.org/jira/browse/SPARK-10619\n\nlooks like this was broken by commit: https://github.com/apache/spark/commit/fb1d06fc242ec00320f1a3049673fbb03c4a6eb9#diff-b8adb646ef90f616c34eb5c98d1ebd16\nIt looks like somethings were change to use the UIUtils.listingTable but executor page wasn't converted so when it removed sortable from the UIUtils. TABLE_CLASS_NOT_STRIPED it broke this page.\n\nSimply add the sortable tag back in and it fixes both active UI and the history server UI.\n\nAuthor: Tom Graves <tgraves@yahoo-inc.com>\n\nCloses #9101 from tgravescs/SPARK-10619.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/135a2ce5b0b927b512c832d61c25e7b9d57e30be"},{"sha":"31f315981709251d5d26c508a3dc62cf0e6f87e1","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-11040] [NETWORK] Make sure SASL handler delegates all events.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #9053 from vanzin/SPARK-11040.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/31f315981709251d5d26c508a3dc62cf0e6f87e1"},{"sha":"7e1308d37f6ca35f063e67e4b87a77e932ad89a5","author":{"email":"b1011540267ee7fe5a02243ad09c007ca1860809@us.ibm.com","name":"Huaxin Gao"},"message":"[SPARK-8386] [SQL] add write.mode for insertIntoJDBC when the parm overwrite is false\n\nthe fix is for jira https://issues.apache.org/jira/browse/SPARK-8386\n\nAuthor: Huaxin Gao <huaxing@us.ibm.com>\n\nCloses #9042 from huaxingao/spark8386.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/7e1308d37f6ca35f063e67e4b87a77e932ad89a5"},{"sha":"615cc858cf913522059b6ebdde65f0204f4fb030","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-10973]\n\nClose #9064\nClose #9063\nClose #9062\n\nThese pull requests were merged into branch-1.5, branch-1.4, and branch-1.3.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/615cc858cf913522059b6ebdde65f0204f4fb030"},{"sha":"cf2e0ae7205443f052463e8cb9334ae2b6df2d0e","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11096] Post-hoc review Netty based RPC implementation - round 2\n\nA few more changes:\n\n1. Renamed IDVerifier -> RpcEndpointVerifier\n2. Renamed NettyRpcAddress -> RpcEndpointAddress\n3. Simplified NettyRpcHandler a bit by removing the connection count tracking. This is OK because I now force spark.shuffle.io.numConnectionsPerPeer to 1\n4. Reduced spark.rpc.connect.threads to 64. It would be great to eventually remove this extra thread pool.\n5. Minor cleanup & documentation.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9112 from rxin/SPARK-11096.","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/cf2e0ae7205443f052463e8cb9334ae2b6df2d0e"},{"sha":"239c040414b95d2f8af6e7af46d4832b3f2164b2","author":{"email":"5606b020d06c40eebc00f1e42d81d8836f0a1591@163.com","name":"fenghaixiong"},"message":"Merge remote branch 'upstream/master'","distinct":true,"url":"https://api.github.com/repos/hxfeng/spark/commits/239c040414b95d2f8af6e7af46d4832b3f2164b2"}]}