{"push_id":824688625,"size":6,"distinct_size":6,"ref":"refs/heads/master","head":"c4da5345a0ef643a7518756caaa18ff3f3ea9acc","before":"64b1d00e1a7c1dc52c08a5e97baf6e7117f1a94f","commits":[{"sha":"149472a01d12828c64b0a852982d48c123984182","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-11023] [YARN] Avoid creating URIs from local paths directly.\n\nThe issue is that local paths on Windows, when provided with drive\nletters or backslashes, are not valid URIs.\n\nInstead of trying to figure out whether paths are URIs or not, use\nUtils.resolveURI() which does that for us.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #9049 from vanzin/SPARK-11023 and squashes the following commits:\n\n77021f2 [Marcelo Vanzin] [SPARK-11023] [yarn] Avoid creating URIs from local paths directly.","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/149472a01d12828c64b0a852982d48c123984182"},{"sha":"2e572c4135c3f5ad3061c1f58cdb8a70bed0a9d3","author":{"email":"353f486bccb994414888bc877b93c9c72986cfb3@netflix.com","name":"Ashwin Shankar"},"message":"[SPARK-8170] [PYTHON] Add signal handler to trap Ctrl-C in pyspark and cancel all running jobs\n\nThis patch adds a signal handler to trap Ctrl-C and cancels running job.\n\nAuthor: Ashwin Shankar <ashankar@netflix.com>\n\nCloses #9033 from ashwinshankar77/master.","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/2e572c4135c3f5ad3061c1f58cdb8a70bed0a9d3"},{"sha":"8a354bef55ce9cc0fa77fa1c3a9d62c16438ca1b","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-11042] [SQL] Add a mechanism to ban creating multiple root SQLContexts/HiveContexts in a JVM\n\nhttps://issues.apache.org/jira/browse/SPARK-11042\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #9058 from yhuai/SPARK-11042.","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/8a354bef55ce9cc0fa77fa1c3a9d62c16438ca1b"},{"sha":"091c2c3ecd69803d78c2b15a1487046701059d38","author":{"email":"284a00ea9901227c42561196ff8576aed3ac5631@gmail.com","name":"Kay Ousterhout"},"message":"[SPARK-11056] Improve documentation of SBT build.\n\nThis commit improves the documentation around building Spark to\n(1) recommend using SBT interactive mode to avoid the overhead of\nlaunching SBT and (2) refer to the wiki page that documents using\nSPARK_PREPEND_CLASSES to avoid creating the assembly jar for each\ncompile.\n\ncc srowen\n\nAuthor: Kay Ousterhout <kayousterhout@gmail.com>\n\nCloses #9068 from kayousterhout/SPARK-11056.","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/091c2c3ecd69803d78c2b15a1487046701059d38"},{"sha":"f97e9323b526b3d0b0fee0ca03f4276f37bb5750","author":{"email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com","name":"jerryshao"},"message":"[SPARK-10739] [YARN] Add application attempt window for Spark on Yarn\n\nAdd application attempt window for Spark on Yarn to ignore old out of window failures, this is useful for long running applications to recover from failures.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #8857 from jerryshao/SPARK-10739 and squashes the following commits:\n\n36eabdc [jerryshao] change the doc\n7f9b77d [jerryshao] Style change\n1c9afd0 [jerryshao] Address the comments\ncaca695 [jerryshao] Add application attempt window for Spark on Yarn","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/f97e9323b526b3d0b0fee0ca03f4276f37bb5750"},{"sha":"c4da5345a0ef643a7518756caaa18ff3f3ea9acc","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-10990] [SPARK-11018] [SQL] improve unrolling of complex types\n\nThis PR improve the unrolling and read of complex types in columnar cache:\n1) Using UnsafeProjection to do serialization of complex types, so they will not be serialized three times (two for actualSize)\n2) Copy the bytes from UnsafeRow/UnsafeArrayData to ByteBuffer directly, avoiding the immediate byte[]\n3) Using the underlying array in ByteBuffer to create UTF8String/UnsafeRow/UnsafeArrayData without copy.\n\nCombine these optimizations,  we can reduce the unrolling time from 25s to 21s (20% less), reduce the scanning time from 3.5s to 2.5s (28% less).\n\n```\ndf = sqlContext.read.parquet(path)\nt = time.time()\ndf.cache()\ndf.count()\nprint 'unrolling', time.time() - t\n\nfor i in range(10):\n    t = time.time()\n    print df.select(\"*\")._jdf.queryExecution().toRdd().count()\n    print time.time() - t\n```\n\nThe schema is\n```\nroot\n |-- a: struct (nullable = true)\n |    |-- b: long (nullable = true)\n |    |-- c: string (nullable = true)\n |-- d: array (nullable = true)\n |    |-- element: long (containsNull = true)\n |-- e: map (nullable = true)\n |    |-- key: long\n |    |-- value: string (valueContainsNull = true)\n```\n\nNow the columnar cache depends on that UnsafeProjection support all the data types (including UDT), this PR also fix that.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #9016 from davies/complex2.","distinct":true,"url":"https://api.github.com/repos/jkbradley/spark/commits/c4da5345a0ef643a7518756caaa18ff3f3ea9acc"}]}