{"push_id":827540668,"size":33,"distinct_size":31,"ref":"refs/heads/SPARK-6488_Add_Addition_and_Multiplication_to_PySpark_BlockMatrix","head":"67a5b781cb98e0ffec1f74b98e483a64644e654f","before":"142b95ee821ffafc3bcc638189436c7e24fb4309","commits":[{"sha":"8a354bef55ce9cc0fa77fa1c3a9d62c16438ca1b","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-11042] [SQL] Add a mechanism to ban creating multiple root SQLContexts/HiveContexts in a JVM\n\nhttps://issues.apache.org/jira/browse/SPARK-11042\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #9058 from yhuai/SPARK-11042.","distinct":false,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/8a354bef55ce9cc0fa77fa1c3a9d62c16438ca1b"},{"sha":"091c2c3ecd69803d78c2b15a1487046701059d38","author":{"email":"284a00ea9901227c42561196ff8576aed3ac5631@gmail.com","name":"Kay Ousterhout"},"message":"[SPARK-11056] Improve documentation of SBT build.\n\nThis commit improves the documentation around building Spark to\n(1) recommend using SBT interactive mode to avoid the overhead of\nlaunching SBT and (2) refer to the wiki page that documents using\nSPARK_PREPEND_CLASSES to avoid creating the assembly jar for each\ncompile.\n\ncc srowen\n\nAuthor: Kay Ousterhout <kayousterhout@gmail.com>\n\nCloses #9068 from kayousterhout/SPARK-11056.","distinct":false,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/091c2c3ecd69803d78c2b15a1487046701059d38"},{"sha":"f97e9323b526b3d0b0fee0ca03f4276f37bb5750","author":{"email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com","name":"jerryshao"},"message":"[SPARK-10739] [YARN] Add application attempt window for Spark on Yarn\n\nAdd application attempt window for Spark on Yarn to ignore old out of window failures, this is useful for long running applications to recover from failures.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #8857 from jerryshao/SPARK-10739 and squashes the following commits:\n\n36eabdc [jerryshao] change the doc\n7f9b77d [jerryshao] Style change\n1c9afd0 [jerryshao] Address the comments\ncaca695 [jerryshao] Add application attempt window for Spark on Yarn","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/f97e9323b526b3d0b0fee0ca03f4276f37bb5750"},{"sha":"c4da5345a0ef643a7518756caaa18ff3f3ea9acc","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-10990] [SPARK-11018] [SQL] improve unrolling of complex types\n\nThis PR improve the unrolling and read of complex types in columnar cache:\n1) Using UnsafeProjection to do serialization of complex types, so they will not be serialized three times (two for actualSize)\n2) Copy the bytes from UnsafeRow/UnsafeArrayData to ByteBuffer directly, avoiding the immediate byte[]\n3) Using the underlying array in ByteBuffer to create UTF8String/UnsafeRow/UnsafeArrayData without copy.\n\nCombine these optimizations,  we can reduce the unrolling time from 25s to 21s (20% less), reduce the scanning time from 3.5s to 2.5s (28% less).\n\n```\ndf = sqlContext.read.parquet(path)\nt = time.time()\ndf.cache()\ndf.count()\nprint 'unrolling', time.time() - t\n\nfor i in range(10):\n    t = time.time()\n    print df.select(\"*\")._jdf.queryExecution().toRdd().count()\n    print time.time() - t\n```\n\nThe schema is\n```\nroot\n |-- a: struct (nullable = true)\n |    |-- b: long (nullable = true)\n |    |-- c: string (nullable = true)\n |-- d: array (nullable = true)\n |    |-- element: long (containsNull = true)\n |-- e: map (nullable = true)\n |    |-- key: long\n |    |-- value: string (valueContainsNull = true)\n```\n\nNow the columnar cache depends on that UnsafeProjection support all the data types (including UDT), this PR also fix that.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #9016 from davies/complex2.","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/c4da5345a0ef643a7518756caaa18ff3f3ea9acc"},{"sha":"626aab79c9b4d4ac9d65bf5fa45b81dd9cbc609c","author":{"email":"0da04896db04dd2393029a4b0c77df7858a20dea@gmail.com","name":"Lianhui Wang"},"message":"[SPARK-11026] [YARN] spark.yarn.user.classpath.first does work for 'spark-submit --jars hdfs://user/foo.jar'\n\nwhen spark.yarn.user.classpath.first=true and using 'spark-submit --jars hdfs://user/foo.jar', it can not put foo.jar to system classpath. so we need to put yarn's linkNames of jars to the system classpath. vanzin tgravescs\n\nAuthor: Lianhui Wang <lianhuiwang09@gmail.com>\n\nCloses #9045 from lianhuiwang/spark-11026.","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/626aab79c9b4d4ac9d65bf5fa45b81dd9cbc609c"},{"sha":"6987c067937a50867b4d5788f5bf496ecdfdb62c","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-11009] [SQL] fix wrong result of Window function in cluster mode\n\nCurrently, All windows function could generate wrong result in cluster sometimes.\n\nThe root cause is that AttributeReference is called in executor, then id of it may not be unique than others created in driver.\n\nHere is the script that could reproduce the problem (run in local cluster):\n```\nfrom pyspark import SparkContext, HiveContext\nfrom pyspark.sql.window import Window\nfrom pyspark.sql.functions import rowNumber\n\nsqlContext = HiveContext(SparkContext())\nsqlContext.setConf(\"spark.sql.shuffle.partitions\", \"3\")\ndf =  sqlContext.range(1<<20)\ndf2 = df.select((df.id % 1000).alias(\"A\"), (df.id / 1000).alias('B'))\nws = Window.partitionBy(df2.A).orderBy(df2.B)\ndf3 = df2.select(\"client\", \"date\", rowNumber().over(ws).alias(\"rn\")).filter(\"rn < 0\")\nassert df3.count() == 0\n```\n\nAuthor: Davies Liu <davies@databricks.com>\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #9050 from davies/wrong_window.","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/6987c067937a50867b4d5788f5bf496ecdfdb62c"},{"sha":"1797055dbf1d2fd7714d7c65c8d2efde2f15efc1","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-11079] Post-hoc review Netty-based RPC - round 1\n\nI'm going through the implementation right now for post-doc review. Adding more comments and renaming things as I go through them.\n\nI also want to write higher level documentation about how the whole thing works -- but those will come in other pull requests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #9091 from rxin/rpc-review.","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/1797055dbf1d2fd7714d7c65c8d2efde2f15efc1"},{"sha":"d0cc79ccd0b4500bd6b18184a723dabc164e8abd","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-11030] [SQL] share the SQLTab across sessions\n\nThe SQLTab will be shared by multiple sessions.\n\nIf we create multiple independent SQLContexts (not using newSession()), will still see multiple SQLTabs in the Spark UI.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #9048 from davies/sqlui.","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/d0cc79ccd0b4500bd6b18184a723dabc164e8abd"},{"sha":"5e3868ba139f5f0b3a33361c6b884594a3ab6421","author":{"email":"3f27eab2afd49386f0ac811afb0bc245aa92c0fa@intel.com","name":"Sun Rui"},"message":"[SPARK-10051] [SPARKR] Support collecting data of StructType in DataFrame\n\nTwo points in this PR:\n\n1.    Originally thought was that a named R list is assumed to be a struct in SerDe. But this is problematic because some R functions will implicitly generate named lists that are not intended to be a struct when transferred by SerDe. So SerDe clients have to explicitly mark a names list as struct by changing its class from \"list\" to \"struct\".\n\n2.    SerDe is in the Spark Core module, and data of StructType is represented as GenricRow which is defined in Spark SQL module. SerDe can't import GenricRow as in maven build  Spark SQL module depends on Spark Core module. So this PR adds a registration hook in SerDe to allow SQLUtils in Spark SQL module to register its functions for serialization and deserialization of StructType.\n\nAuthor: Sun Rui <rui.sun@intel.com>\n\nCloses #8794 from sun-rui/SPARK-10051.","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/5e3868ba139f5f0b3a33361c6b884594a3ab6421"},{"sha":"1e0aba90b9e73834af70d196f7f869b062d98d94","author":{"email":"24293fb0f3748cb0e5fccba5c0446bb95db892a1@gmail.com","name":"Narine Kokhlikyan"},"message":"[SPARK-10888] [SPARKR] Added as.DataFrame as a synonym to createDataFrame\n\nas.DataFrame is more a R-style like signature.\nAlso, I'd like to know if we could make the context, e.g. sqlContext global, so that we do not have to specify it as an argument, when we each time create a dataframe.\n\nAuthor: Narine Kokhlikyan <narine.kokhlikyan@gmail.com>\n\nCloses #8952 from NarineK/sparkrasDataFrame.","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/1e0aba90b9e73834af70d196f7f869b062d98d94"},{"sha":"f7f28ee7a513c262d52cf433d25fbf06df9bd1f1","author":{"email":"cebbefecc4e6d7f2be0a1864a6aa073354240fe8@users.noreply.github.com","name":"Adrian Zhuang"},"message":"[SPARK-10913] [SPARKR] attach() function support\n\nBring the change code up to date.\n\nAuthor: Adrian Zhuang <adrian555@users.noreply.github.com>\nAuthor: adrian555 <wzhuang@us.ibm.com>\n\nCloses #9031 from adrian555/attach2.","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/f7f28ee7a513c262d52cf433d25fbf06df9bd1f1"},{"sha":"c75f058b72d492d6de898957b3058f242d70dd8a","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[PYTHON] [MINOR] List modules in PySpark tests when given bad name\n\nOutput list of supported modules for python tests in error message when given bad module name.\n\nCC: davies\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #9088 from jkbradley/python-tests-modules.","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/c75f058b72d492d6de898957b3058f242d70dd8a"},{"sha":"2b574f52d7bf51b1fe2a73086a3735b633e9083f","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-7402] [ML] JSON SerDe for standard param types\n\nThis PR implements the JSON SerDe for the following param types: `Boolean`, `Int`, `Long`, `Float`, `Double`, `String`, `Array[Int]`, `Array[Double]`, and `Array[String]`. The implementation of `Float`, `Double`, and `Array[Double]` are specialized to handle `NaN` and `Inf`s. This will be used in pipeline persistence. jkbradley\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #9090 from mengxr/SPARK-7402.","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/2b574f52d7bf51b1fe2a73086a3735b633e9083f"},{"sha":"b3ffac5178795f2d8e7908b3e77e8e89f50b5f6f","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-10983] Unified memory manager\n\nThis patch unifies the memory management of the storage and execution regions such that either side can borrow memory from each other. When memory pressure arises, storage will be evicted in favor of execution. To avoid regressions in cases where storage is crucial, we dynamically allocate a fraction of space for storage that execution cannot evict. Several configurations are introduced:\n\n- **spark.memory.fraction (default 0.75)**: ​fraction of the heap space used for execution and storage. The lower this is, the more frequently spills and cached data eviction occur. The purpose of this config is to set aside memory for internal metadata, user data structures, and imprecise size estimation in the case of sparse, unusually large records.\n\n- **spark.memory.storageFraction (default 0.5)**: size of the storage region within the space set aside by `s​park.memory.fraction`. ​Cached data may only be evicted if total storage exceeds this region.\n\n- **spark.memory.useLegacyMode (default false)**: whether to use the memory management that existed in Spark 1.5 and before. This is mainly for backward compatibility.\n\nFor a detailed description of the design, see [SPARK-10000](https://issues.apache.org/jira/browse/SPARK-10000). This patch builds on top of the `MemoryManager` interface introduced in #9000.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #9084 from andrewor14/unified-memory-manager.","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/b3ffac5178795f2d8e7908b3e77e8e89f50b5f6f"},{"sha":"0d1b73b78b600420121ea8e58ff659ae8b4feebe","author":{"email":"0365f6b3a191ca1f960a8b40456e71c9e5c1e128@atscale.com","name":"trystanleftwich"},"message":"[SPARK-11052] Spaces in the build dir causes failures in the build/mv…\n\n…n script\n\nAuthor: trystanleftwich <trystan@atscale.com>\n\nCloses #9065 from trystanleftwich/SPARK-11052.","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/0d1b73b78b600420121ea8e58ff659ae8b4feebe"},{"sha":"ef72673b234579c161b8cbb6cafc851d9eba1bfb","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-11080] [SQL] Incorporate per-JVM id into ExprId to prevent unsafe cross-JVM comparisions\n\nIn the current implementation of named expressions' `ExprIds`, we rely on a per-JVM AtomicLong to ensure that expression ids are unique within a JVM. However, these expression ids will not be _globally_ unique. This opens the potential for id collisions if new expression ids happen to be created inside of tasks rather than on the driver.\n\nThere are currently a few cases where tasks allocate expression ids, which happen to be safe because those expressions are never compared to expressions created on the driver. In order to guard against the introduction of invalid comparisons between driver-created and executor-created expression ids, this patch extends `ExprId` to incorporate a UUID to identify the JVM that created the id, which prevents collisions.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9093 from JoshRosen/SPARK-11080.","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/ef72673b234579c161b8cbb6cafc851d9eba1bfb"},{"sha":"d0482f6af33e976db237405b2a978db1b7c2fd5b","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-10932] [PROJECT INFRA] Port two minor changes to release-build.sh from scripts' old repo\n\nSpark's release packaging scripts used to live in a separate repository. Although these scripts are now part of the Spark repo, there are some minor patches made against the old repos that are missing in Spark's copy of the script. This PR ports those changes.\n\n/cc shivaram, who originally submitted these changes against https://github.com/rxin/spark-utils\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #8986 from JoshRosen/port-release-build-fixes-from-rxin-repo.","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/d0482f6af33e976db237405b2a978db1b7c2fd5b"},{"sha":"3889b1c7a96da1111946fa63ad69489b83468646","author":{"email":"a8225d365811cb0449b395490c087d0ddd3c6c6f@gmail.com","name":"vectorijk"},"message":"[SPARK-11059] [ML] Change range of quantile probabilities in AFTSurvivalRegression\n\nValue of the quantile probabilities array should be in the range (0, 1) instead of [0,1]\n in `AFTSurvivalRegression.scala` according to [Discussion] (https://github.com/apache/spark/pull/8926#discussion-diff-40698242)\n\nAuthor: vectorijk <jiangkai@gmail.com>\n\nCloses #9083 from vectorijk/spark-11059.","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/3889b1c7a96da1111946fa63ad69489b83468646"},{"sha":"328d1b3e4bc39cce653342e04f9e08af12dd7ed8","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-11090] [SQL] Constructor for Product types from InternalRow\n\nThis is a first draft of the ability to construct expressions that will take a catalyst internal row and construct a Product (case class or tuple) that has fields with the correct names.  Support include:\n - Nested classes\n - Maps\n - Efficiently handling of arrays of primitive types\n\nNot yet supported:\n - Case classes that require custom collection types (i.e. List instead of Seq).\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #9100 from marmbrus/productContructor.","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/328d1b3e4bc39cce653342e04f9e08af12dd7ed8"},{"sha":"e170c22160bb452f98c340489ebf8390116a8cbb","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@163.com","name":"Wenchen Fan"},"message":"[SPARK-11032] [SQL] correctly handle having\n\nWe should not stop resolving having when the having condtion is resolved, or something like `count(1)` will crash.\n\nAuthor: Wenchen Fan <cloud0fan@163.com>\n\nCloses #9105 from cloud-fan/having.","distinct":true,"url":"https://api.github.com/repos/dusenberrymw/spark/commits/e170c22160bb452f98c340489ebf8390116a8cbb"}]}