{"push_id":622966852,"size":986,"distinct_size":986,"ref":"refs/heads/master","head":"b52c7f9fc87a1b9a039724e1dac8b30554f75196","before":"39e333ec4350ddafe29ee0958c37eec07bec85df","commits":[{"sha":"8ead999fd627b12837fb2f082a0e76e9d121d269","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-5223] [MLlib] [PySpark] fix MapConverter and ListConverter in MLlib\n\nIt will introduce problems if the object in dict/list/tuple can not support by py4j, such as Vector.\nAlso, pickle may have better performance for larger object (less RPC).\n\nIn some cases that the object in dict/list can not be pickled (such as JavaObject), we should still use MapConvert/ListConvert.\n\nThis PR should be ported into branch-1.2\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #4023 from davies/listconvert and squashes the following commits:\n\n55d4ab2 [Davies Liu] fix MapConverter and ListConverter in MLlib","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/8ead999fd627b12837fb2f082a0e76e9d121d269"},{"sha":"6463e0b9e8067cce70602c5c9006a2546856a9d6","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-4912][SQL] Persistent tables for the Spark SQL data sources api\n\nWith changes in this PR, users can persist metadata of tables created based on the data source API in metastore through DDLs.\n\nAuthor: Yin Huai <yhuai@databricks.com>\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #3960 from yhuai/persistantTablesWithSchema2 and squashes the following commits:\n\n069c235 [Yin Huai] Make exception messages user friendly.\nc07cbc6 [Yin Huai] Get the location of test file in a correct way.\n4456e98 [Yin Huai] Test data.\n5315dfc [Yin Huai] rxin's comments.\n7fc4b56 [Yin Huai] Add DDLStrategy and HiveDDLStrategy to plan DDLs based on the data source API.\naeaf4b3 [Yin Huai] Add comments.\n06f9b0c [Yin Huai] Revert unnecessary changes.\nfeb88aa [Yin Huai] Merge remote-tracking branch 'apache/master' into persistantTablesWithSchema2\n172db80 [Yin Huai] Fix unit test.\n49bf1ac [Yin Huai] Unit tests.\n8f8f1a1 [Yin Huai] [SPARK-4574][SQL] Adding support for defining schema in foreign DDL commands. #3431\nf47fda1 [Yin Huai] Unit tests.\n2b59723 [Michael Armbrust] Set external when creating tables\nc00bb1b [Michael Armbrust] Don't use reflection to read options\n1ea6e7b [Michael Armbrust] Don't fail when trying to uncache a table that doesn't exist\n6edc710 [Michael Armbrust] Add tests.\nd7da491 [Michael Armbrust] First draft of persistent tables.","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/6463e0b9e8067cce70602c5c9006a2546856a9d6"},{"sha":"14e3f114efb906937b2d7b7ac04484b2814a3b48","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-5168] Make SQLConf a field rather than mixin in SQLContext\n\nThis change should be binary and source backward compatible since we didn't change any user facing APIs.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #3965 from rxin/SPARK-5168-sqlconf and squashes the following commits:\n\n42eec09 [Reynold Xin] Fix default conf value.\n0ef86cc [Reynold Xin] Fix constructor ordering.\n4d7f910 [Reynold Xin] Properly override config.\nccc8e6a [Reynold Xin] [SPARK-5168] Make SQLConf a field rather than mixin in SQLContext","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/14e3f114efb906937b2d7b7ac04484b2814a3b48"},{"sha":"f9969098c8cb15e36c718b80c6cf5b534a6cf7c3","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-5123][SQL] Reconcile Java/Scala API for data types.\n\nHaving two versions of the data type APIs (one for Java, one for Scala) requires downstream libraries to also have two versions of the APIs if the library wants to support both Java and Scala. I took a look at the Scala version of the data type APIs - it can actually work out pretty well for Java out of the box.\n\nAs part of the PR, I created a sql.types package and moved all type definitions there. I then removed the Java specific data type API along with a lot of the conversion code.\n\nThis subsumes https://github.com/apache/spark/pull/3925\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #3958 from rxin/SPARK-5123-datatype-2 and squashes the following commits:\n\n66505cc [Reynold Xin] [SPARK-5123] Expose only one version of the data type APIs (i.e. remove the Java-specific API).","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/f9969098c8cb15e36c718b80c6cf5b534a6cf7c3"},{"sha":"d5eeb35167e1ab72fab7778757163ff0aacaef2c","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-5167][SQL] Move Row into sql package and make it usable for Java.\n\nMostly just moving stuff around. This should still be source compatible since we type aliased Row previously in org.apache.spark.sql.Row.\n\nAdded the following APIs to Row:\n```scala\ndef getMap[K, V](i: Int): scala.collection.Map[K, V]\ndef getJavaMap[K, V](i: Int): java.util.Map[K, V]\ndef getSeq[T](i: Int): Seq[T]\ndef getList[T](i: Int): java.util.List[T]\ndef getStruct(i: Int): StructType\n```\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #4030 from rxin/sql-row and squashes the following commits:\n\n6c85c29 [Reynold Xin] Fixed style violation by adding a new line to Row.scala.\n82b064a [Reynold Xin] [SPARK-5167][SQL] Move Row into sql package and make it usable for Java.","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/d5eeb35167e1ab72fab7778757163ff0aacaef2c"},{"sha":"a3f7421b42f45e39f3e53679188e4eae2ed1f208","author":{"email":"c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","name":"Daoyuan Wang"},"message":"[SPARK-5248] [SQL] move sql.types.decimal.Decimal to sql.types.Decimal\n\nrxin follow up of #3732\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #4041 from adrian-wang/decimal and squashes the following commits:\n\naa3d738 [Daoyuan Wang] fix auto refactor\n7777a58 [Daoyuan Wang] move sql.types.decimal.Decimal to sql.types.Decimal","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/a3f7421b42f45e39f3e53679188e4eae2ed1f208"},{"sha":"81f72a0df2250debe8a6a0773d809d8c42eeabb9","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-5211][SQL]Restore HiveMetastoreTypes.toDataType\n\njira: https://issues.apache.org/jira/browse/SPARK-5211\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #4026 from yhuai/SPARK-5211 and squashes the following commits:\n\n15ee32b [Yin Huai] Remove extra line.\nc6c1651 [Yin Huai] Get back HiveMetastoreTypes.toDataType.","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/81f72a0df2250debe8a6a0773d809d8c42eeabb9"},{"sha":"38bdc992a1a0485ac630af500da54f0a77e133bf","author":{"email":"c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","name":"Daoyuan Wang"},"message":"[SQL] some comments fix for GROUPING SETS\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #4000 from adrian-wang/comment and squashes the following commits:\n\n9c24fc4 [Daoyuan Wang] some comments","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/38bdc992a1a0485ac630af500da54f0a77e133bf"},{"sha":"5840f5464bad8431810d459c97d6e4635eea175c","author":{"email":"8757c4a255384de9fc2175f17a84f948e7081d30@gmail.com","name":"MechCoder"},"message":"[SPARK-2909] [MLlib] [PySpark] SparseVector in pyspark now supports indexing\n\nSlightly different than the scala code which converts the sparsevector into a densevector and then checks the index.\n\nI also hope I've added tests in the right place.\n\nAuthor: MechCoder <manojkumarsivaraj334@gmail.com>\n\nCloses #4025 from MechCoder/spark-2909 and squashes the following commits:\n\n07d0f26 [MechCoder] STY: Rename item to index\nf02148b [MechCoder] [SPARK-2909] [Mlib] SparseVector in pyspark now supports indexing","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/5840f5464bad8431810d459c97d6e4635eea175c"},{"sha":"9d4449c4b3d0f5e60e71fef3a9b2e1c58a80b9e8","author":{"email":"7d45abe39fd242ee2f6575da28f13d66568418b3@oss.nttdata.co.jp","name":"Kousuke Saruta"},"message":"[SPARK-5228][WebUI] Hide tables for \"Active Jobs/Completed Jobs/Failed Jobs\" when they are empty\n\nIn current WebUI, tables for Active Stages, Completed Stages, Skipped Stages and Failed Stages are hidden when they are empty while tables for Active Jobs, Completed Jobs and Failed Jobs are not hidden though they are empty.\n\nThis is before my patch is applied.\n\n![2015-01-13 14 13 03](https://cloud.githubusercontent.com/assets/4736016/5730793/2b73d6f4-9b32-11e4-9a24-1784d758c644.png)\n\nAnd this is after my patch is applied.\n\n![2015-01-13 14 38 13](https://cloud.githubusercontent.com/assets/4736016/5730797/359ea2da-9b32-11e4-97b0-544739ddbf4c.png)\n\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #4028 from sarutak/SPARK-5228 and squashes the following commits:\n\nb1e6e8b [Kousuke Saruta] Fixed a small typo\ndaab563 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-5228\n9493a1d [Kousuke Saruta] Modified AllJobPage.scala so that hide Active Jobs/Completed Jobs/Failed Jobs when they are empty","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/9d4449c4b3d0f5e60e71fef3a9b2e1c58a80b9e8"},{"sha":"259936be710f367e1d1e019ee7b93fb68dfc33d0","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-4014] Add TaskContext.attemptNumber and deprecate TaskContext.attemptId\n\n`TaskContext.attemptId` is misleadingly-named, since it currently returns a taskId, which uniquely identifies a particular task attempt within a particular SparkContext, instead of an attempt number, which conveys how many times a task has been attempted.\n\nThis patch deprecates `TaskContext.attemptId` and add `TaskContext.taskId` and `TaskContext.attemptNumber` fields.  Prior to this change, it was impossible to determine whether a task was being re-attempted (or was a speculative copy), which made it difficult to write unit tests for tasks that fail on early attempts or speculative tasks that complete faster than original tasks.\n\nEarlier versions of the TaskContext docs suggest that `attemptId` behaves like `attemptNumber`, so there's an argument to be made in favor of changing this method's implementation.  Since we've decided against making that change in maintenance branches, I think it's simpler to add better-named methods and retain the old behavior for `attemptId`; if `attemptId` behaved differently in different branches, then this would cause confusing build-breaks when backporting regression tests that rely on the new `attemptId` behavior.\n\nMost of this patch is fairly straightforward, but there is a bit of trickiness related to Mesos tasks: since there's no field in MesosTaskInfo to encode the attemptId, I packed it into the `data` field alongside the task binary.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #3849 from JoshRosen/SPARK-4014 and squashes the following commits:\n\n89d03e0 [Josh Rosen] Merge remote-tracking branch 'origin/master' into SPARK-4014\n5cfff05 [Josh Rosen] Introduce wrapper for serializing Mesos task launch data.\n38574d4 [Josh Rosen] attemptId -> taskAttemptId in PairRDDFunctions\na180b88 [Josh Rosen] Merge remote-tracking branch 'origin/master' into SPARK-4014\n1d43aa6 [Josh Rosen] Merge remote-tracking branch 'origin/master' into SPARK-4014\neee6a45 [Josh Rosen] Merge remote-tracking branch 'origin/master' into SPARK-4014\n0b10526 [Josh Rosen] Use putInt instead of putLong (silly mistake)\n8c387ce [Josh Rosen] Use local with maxRetries instead of local-cluster.\ncbe4d76 [Josh Rosen] Preserve attemptId behavior and deprecate it:\nb2dffa3 [Josh Rosen] Address some of Reynold's minor comments\n9d8d4d1 [Josh Rosen] Doc typo\n1e7a933 [Josh Rosen] [SPARK-4014] Change TaskContext.attemptId to return attempt number instead of task ID.\nfd515a5 [Josh Rosen] Add failing test for SPARK-4014","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/259936be710f367e1d1e019ee7b93fb68dfc33d0"},{"sha":"2fd7f72b6b0b24bec12331c7bbbcf6bfc265d2ec","author":{"email":"d1bfea4c6b6818449d80331b0fceaa05c11d6f50@gmail.com","name":"Alex Baretta"},"message":"[SPARK-5235] Make SQLConf Serializable\n\nDeclare SQLConf to be serializable to fix \"Task not serializable\" exceptions in SparkSQL\n\nAuthor: Alex Baretta <alexbaretta@gmail.com>\n\nCloses #4031 from alexbaretta/SPARK-5235-SQLConf and squashes the following commits:\n\nc2103f5 [Alex Baretta] [SPARK-5235] Make SQLConf Serializable","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/2fd7f72b6b0b24bec12331c7bbbcf6bfc265d2ec"},{"sha":"76389c5b99183e456ff85fd92ea68d95c4c13e82","author":{"email":"eb099f5da6a5c736bf79e31da873c42fd6bc909b@yuhaodevbox.sh.intel.com","name":"Yuhao Yang"},"message":"[SPARK-5234][ml]examples for ml don't have sparkContext.stop\n\nJIRA issue: https://issues.apache.org/jira/browse/SPARK-5234\n\nsimply add the call.\n\nAuthor: Yuhao Yang <yuhao@yuhaodevbox.sh.intel.com>\n\nCloses #4044 from hhbyyh/addscStop and squashes the following commits:\n\nc1f75ac [Yuhao Yang] add SparkContext.stop to 3 ml examples","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/76389c5b99183e456ff85fd92ea68d95c4c13e82"},{"sha":"13d2406781714daea2bbf3bfb7fec0dead10760c","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-5254][MLLIB] Update the user guide to position spark.ml better\n\nThe current statement in the user guide may deliver confusing messages to users. spark.ml contains high-level APIs for building ML pipelines. But it doesn't mean that spark.mllib is being deprecated.\n\nFirst of all, the pipeline API is in its alpha stage and we need to see more use cases from the community to stabilizes it, which may take several releases. Secondly, the components in spark.ml are simple wrappers over spark.mllib implementations. Neither the APIs or the implementations from spark.mllib are being deprecated. We expect users use spark.ml pipeline APIs to build their ML pipelines, but we will keep supporting and adding features to spark.mllib. For example, there are many features in review at https://spark-prs.appspot.com/#mllib. So users should be comfortable with using spark.mllib features and expect more coming. The user guide needs to be updated to make the message clear.\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #4052 from mengxr/SPARK-5254 and squashes the following commits:\n\n6d5f1d3 [Xiangrui Meng] typo\n0cc935b [Xiangrui Meng] update user guide to position spark.ml better","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/13d2406781714daea2bbf3bfb7fec0dead10760c"},{"sha":"cfa397c126c857bfc9843d9e598a14b7c1e0457f","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-5193][SQL] Tighten up SQLContext API\n\n1. Removed 2 implicits (logicalPlanToSparkQuery and baseRelationToSchemaRDD)\n2. Moved extraStrategies into ExperimentalMethods.\n3. Made private methods protected[sql] so they don't show up in javadocs.\n4. Removed createParquetFile.\n5. Added Java version of applySchema to SQLContext.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #4049 from rxin/sqlContext-refactor and squashes the following commits:\n\na326a1a [Reynold Xin] Remove createParquetFile and add applySchema for Java to SQLContext.\necd6685 [Reynold Xin] Added baseRelationToSchemaRDD back.\n4a38c9b [Reynold Xin] [SPARK-5193][SQL] Tighten up SQLContext API","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/cfa397c126c857bfc9843d9e598a14b7c1e0457f"},{"sha":"6abc45e340d3be5f07236adc104db5f8dda0d514","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-5254][MLLIB] remove developers section from spark.ml guide\n\nForgot to remove this section in #4052.\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #4053 from mengxr/SPARK-5254-update and squashes the following commits:\n\nf295bde [Xiangrui Meng] remove developers section from spark.ml guide","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/6abc45e340d3be5f07236adc104db5f8dda0d514"},{"sha":"4b325c77a270ec32d6858d204313d4f161774fae","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-5193][SQL] Tighten up HiveContext API\n\n1. Removed the deprecated LocalHiveContext\n2. Made private[sql] fields protected[sql] so they don't show up in javadoc.\n3. Added javadoc to refreshTable.\n4. Added Experimental tag to analyze command.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #4054 from rxin/hivecontext-api and squashes the following commits:\n\n25cc00a [Reynold Xin] Add implicit conversion back.\ncbca886 [Reynold Xin] [SPARK-5193][SQL] Tighten up HiveContext API","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/4b325c77a270ec32d6858d204313d4f161774fae"},{"sha":"3c8650c12ad7a97852e7bd76153210493fd83e92","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-5224] [PySpark] improve performance of parallelize list/ndarray\n\nAfter the default batchSize changed to 0 (batched based on the size of object), but parallelize() still use BatchedSerializer with batchSize=1, this PR will use batchSize=1024 for parallelize by default.\n\nAlso, BatchedSerializer did not work well with list and numpy.ndarray, this improve BatchedSerializer by using __len__ and __getslice__.\n\nHere is the benchmark for parallelize 1 millions int with list or ndarray:\n\n    |          before     |   after  | improvements\n ------- | ------------ | ------------- | -------\nlist |   11.7 s  | 0.8 s |  14x\nnumpy.ndarray     |  32 s  |   0.7 s | 40x\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #4024 from davies/opt_numpy and squashes the following commits:\n\n7618c7c [Davies Liu] improve performance of parallelize list/ndarray","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/3c8650c12ad7a97852e7bd76153210493fd83e92"},{"sha":"1881431dd50e93a6948e4966d33742727f27e917","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-5274][SQL] Reconcile Java and Scala UDFRegistration.\n\nAs part of SPARK-5193:\n\n1. Removed UDFRegistration as a mixin in SQLContext and made it a field (\"udf\").\n2. For Java UDFs, renamed dataType to returnType.\n3. For Scala UDFs, added type tags.\n4. Added all Java UDF registration methods to Scala's UDFRegistration.\n5. Documentation\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #4056 from rxin/udf-registration and squashes the following commits:\n\nae9c556 [Reynold Xin] Updated example.\n675a3c9 [Reynold Xin] Style fix\n47c24ff [Reynold Xin] Python fix.\n5f00c45 [Reynold Xin] Restore data type position in java udf and added typetags.\n032f006 [Reynold Xin] [SPARK-5193][SQL] Reconcile Java and Scala UDFRegistration.","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/1881431dd50e93a6948e4966d33742727f27e917"},{"sha":"65858ba555c4c0aea988b8f4b1c8476c42377eb9","author":{"email":"7d45abe39fd242ee2f6575da28f13d66568418b3@oss.nttdata.co.jp","name":"Kousuke Saruta"},"message":"[Minor] Fix tiny typo in BlockManager\n\nIn BlockManager, there is a word `BlockTranserService` but I think it's typo for `BlockTransferService`.\n\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #4046 from sarutak/fix-tiny-typo and squashes the following commits:\n\na3e2a2f [Kousuke Saruta] Fixed tiny typo in BlockManager","distinct":true,"url":"https://api.github.com/repos/juanrh/spark/commits/65858ba555c4c0aea988b8f4b1c8476c42377eb9"}]}