{"push_id":555445025,"size":928,"distinct_size":14,"ref":"refs/heads/hbase","head":"011d9dcb2f6f142f2bbfdec5f8a8405f26aa884b","before":"cc84d503c9f9af908479b4bf891b0bfa5fc3ac47","commits":[{"sha":"1b3ce61ce9061719d8c28d621c3200e0bff3a4e9","author":{"email":"a50d2185142f97721f64187b05c8deeec99604b1@gmail.com","name":"Cheng Lian"},"message":"[SPARK-3906][SQL] Adds multiple join support for SQLContext\n\nAuthor: Cheng Lian <lian.cs.zju@gmail.com>\n\nCloses #2767 from liancheng/multi-join and squashes the following commits:\n\n9dc0d18 [Cheng Lian] Adds multiple join support for SQLContext","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/1b3ce61ce9061719d8c28d621c3200e0bff3a4e9"},{"sha":"e9c1afa87b8a6d9ea02aa9f7c319a36f00c0e4c3","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-3800][SQL] Clean aliases from grouping expressions\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #2658 from marmbrus/nestedAggs and squashes the following commits:\n\n862b763 [Michael Armbrust] Merge remote-tracking branch 'origin/master' into nestedAggs\n3234521 [Michael Armbrust] Merge remote-tracking branch 'origin/master' into nestedAggs\n8b06fdc [Michael Armbrust] possible fix for grouping on nested fields","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/e9c1afa87b8a6d9ea02aa9f7c319a36f00c0e4c3"},{"sha":"364d52b707b5c49bc29ce60dbfe6e845a75d5d86","author":{"email":"f070f8f0c50caeda5841963fc7b57e73e59dad56@happy-camper.st","name":"Takuya UESHIN"},"message":"[SPARK-3966][SQL] Fix nullabilities of Cast related to DateType.\n\nAuthor: Takuya UESHIN <ueshin@happy-camper.st>\n\nCloses #2820 from ueshin/issues/SPARK-3966 and squashes the following commits:\n\nca4a745 [Takuya UESHIN] Fix nullabilities of Cast related to DateType.","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/364d52b707b5c49bc29ce60dbfe6e845a75d5d86"},{"sha":"fce1d41611fdb27956c3394a706ed14960182a83","author":{"email":"ae5b4077bc1239c0f5b8129b32bd3d92ff16670c@gmail.com","name":"luogankun"},"message":"[SPARK-3945]Properties of hive-site.xml is invalid in running the Thrift JDBC server\n\nWrite properties of hive-site.xml to HiveContext when initilize session state in SparkSQLEnv.scala.\n\nThe method of SparkSQLEnv.init() in HiveThriftServer2.scala can not write the properties of hive-site.xml to HiveContext. Such as: add configuration property spark.sql.shuffle.partititions in the hive-site.xml.\n\nAuthor: luogankun <luogankun@gmail.com>\n\nCloses #2800 from luogankun/SPARK-3945 and squashes the following commits:\n\n3679efc [luogankun] [SPARK-3945]Write properties of hive-site.xml to HiveContext when initilize session state In SparkSQLEnv.scala","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/fce1d41611fdb27956c3394a706ed14960182a83"},{"sha":"7586e2e67ad45007f78803179b04d199c174bd69","author":{"email":"f070f8f0c50caeda5841963fc7b57e73e59dad56@happy-camper.st","name":"Takuya UESHIN"},"message":"[SPARK-3969][SQL] Optimizer should have a super class as an interface.\n\nSome developers want to replace `Optimizer` to fit their projects but can't do so because currently `Optimizer` is an `object`.\n\nAuthor: Takuya UESHIN <ueshin@happy-camper.st>\n\nCloses #2825 from ueshin/issues/SPARK-3969 and squashes the following commits:\n\nabbc53c [Takuya UESHIN] Re-rename Optimizer object.\n4d2e1bc [Takuya UESHIN] Rename Optimizer object.\n9547a23 [Takuya UESHIN] Extract abstract class from Optimizer for developers to be able to replace Optimizer.","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/7586e2e67ad45007f78803179b04d199c174bd69"},{"sha":"0fe1c093690e5af6268182801747dfa6394dfc5b","author":{"email":"9eee3e63534349a30e59f0f71d1c0ae509ca1e91@gmail.com","name":"wangxiaojing"},"message":"[SPARK-3940][SQL] Avoid console printing error messages three times\n\nIf  wrong sql,the console print error one times。\neg：\n<pre>\nspark-sql> show tabless;\nshow tabless;\n14/10/13 21:03:48 INFO ParseDriver: Parsing command: show tabless\n............\n\tat org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:274)\n\tat org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)\n\tat org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:209)\n\tat org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)\nCaused by: org.apache.hadoop.hive.ql.parse.ParseException: line 1:5 cannot recognize input near 'show' 'tabless' '<EOF>' in ddl statement\n\n\tat org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:193)\n\tat org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:161)\n\tat org.apache.spark.sql.hive.HiveQl$.getAst(HiveQl.scala:218)\n\tat org.apache.spark.sql.hive.HiveQl$.createPlan(HiveQl.scala:226)\n\t... 47 more\nTime taken: 4.35 seconds\n14/10/13 21:03:51 INFO CliDriver: Time taken: 4.35 seconds\n</pre>\n\nAuthor: wangxiaojing <u9jing@gmail.com>\n\nCloses #2790 from wangxiaojing/spark-3940 and squashes the following commits:\n\ne2e5c14 [wangxiaojing] sql Print the error code three times","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/0fe1c093690e5af6268182801747dfa6394dfc5b"},{"sha":"342b57db66e379c475daf5399baf680ff42b87c2","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@apache.org","name":"Reynold Xin"},"message":"Update Building Spark link.","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/342b57db66e379c475daf5399baf680ff42b87c2"},{"sha":"5a8f64f33632fbf89d16cade2e0e66c5ed60760b","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-3958] TorrentBroadcast cleanup / debugging improvements.\n\nThis PR makes several changes to TorrentBroadcast in order to make\nit easier to reason about, which should help when debugging SPARK-3958.\nThe key changes:\n\n- Remove all state from the global TorrentBroadcast object.  This state\n  consisted mainly of configuration options, like the block size and\n  compression codec, and was read by the blockify / unblockify methods.\n  Unfortunately, the use of `lazy val` for `BLOCK_SIZE` meant that the block\n  size was always determined by the first SparkConf that TorrentBroadast was\n  initialized with; as a result, unit tests could not properly test\n  TorrentBroadcast with different block sizes.\n\n  Instead, blockifyObject and unBlockifyObject now accept compression codecs\n  and blockSizes as arguments.  These arguments are supplied at the call sites\n  inside of TorrentBroadcast instances.  Each TorrentBroadcast instance\n  determines these values from SparkEnv's SparkConf.  I was careful to ensure\n  that we do not accidentally serialize CompressionCodec or SparkConf objects\n  as part of the TorrentBroadcast object.\n\n- Remove special-case handling of local-mode in TorrentBroadcast.  I don't\n  think that broadcast implementations should know about whether we're running\n  in local mode.  If we want to optimize the performance of broadcast in local\n  mode, then we should detect this at a higher level and use a dummy\n  LocalBroadcastFactory implementation instead.\n\n  Removing this code fixes a subtle error condition: in the old local mode\n  code, a failure to find the broadcast in the local BlockManager would lead\n  to an attempt to deblockify zero blocks, which could lead to confusing\n  deserialization or decompression errors when we attempted to decompress\n  an empty byte array.  This should never have happened, though: a failure to\n  find the block in local mode is evidence of some other error.  The changes\n  here will make it easier to debug those errors if they ever happen.\n\n- Add a check that throws an exception when attempting to deblockify an\n  empty array.\n\n- Use ScalaCheck to add a test to check that TorrentBroadcast's\n  blockifyObject and unBlockifyObject methods are inverses.\n\n- Misc. cleanup and logging improvements.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #2844 from JoshRosen/torrentbroadcast-bugfix and squashes the following commits:\n\n1e8268d [Josh Rosen] Address Reynold's review comments\n2a9fdfd [Josh Rosen] Address Reynold's review comments.\nc3b08f9 [Josh Rosen] Update TorrentBroadcast tests to reflect removal of special local-mode optimizations.\n5c22782 [Josh Rosen] Store broadcast variable's value in the driver.\n33fc754 [Josh Rosen] Change blockify/unblockifyObject to accept serializer as argument.\n618a872 [Josh Rosen] [SPARK-3958] TorrentBroadcast cleanup / debugging improvements.","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/5a8f64f33632fbf89d16cade2e0e66c5ed60760b"},{"sha":"85708168341a9406c451df20af3374c0850ce166","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-4023] [MLlib] [PySpark] convert rdd into RDD of Vector\n\nConvert the input rdd to RDD of Vector.\n\ncc mengxr\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #2870 from davies/fix4023 and squashes the following commits:\n\n1eac767 [Davies Liu] address comments\n0871576 [Davies Liu] convert rdd into RDD of Vector","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/85708168341a9406c451df20af3374c0850ce166"},{"sha":"2aeb84bc79b643912d26e08ec2d87e444027fefc","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"replace awaitTransformation with awaitTermination in scaladoc/javadoc\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #2861 from holdenk/SPARK-4015-Documentation-in-the-streaming-context-references-non-existent-function and squashes the following commits:\n\n081db8a [Holden Karau] fix pyspark streaming doc too\n0e03863 [Holden Karau] replace awaitTransformation with awaitTermination","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/2aeb84bc79b643912d26e08ec2d87e444027fefc"},{"sha":"c262cd5ddce75333ec936e90b81278c3992841ec","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-4035] Fix a wrong format specifier\n\nJust found a typo. Should not use \"%f\" for Long.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #2875 from zsxwing/SPARK-4035 and squashes the following commits:\n\nce347e2 [zsxwing] Fix a wrong format specifier","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/c262cd5ddce75333ec936e90b81278c3992841ec"},{"sha":"61ca7742d21dd66f5a7b3bb826e3aaca6f049b68","author":{"email":"a5c478d606458f6c868a5126471748acd4730a6f@gmail.com","name":"Andrew Or"},"message":"[SPARK-4020] Do not rely on timeouts to remove failed block managers\n\nIf an executor fails without being scheduled to run any tasks, then `DAGScheduler` won't notify `BlockManagerMasterActor` that the associated block manager should be removed. Instead, the associated block manager will be expired only after a few rounds of heartbeat timeouts. In terms of removal treatment, there should really be no distinction between executors that have been scheduled tasks and those that have not.\n\nThe fix, then, is to add all known executors to `TaskSchedulerImpl`'s `activeExecutorIds` whether or not it has been scheduled a task. In fact, the existing comment above `activeExecutorIds` is\n```\n// Which executor IDs we have executors on\nval activeExecutorIds = new HashSet[String]\n```\nnot  \"Which executors have been scheduled tasks thus far.\"\n\nAuthor: Andrew Or <andrewor14@gmail.com>\n\nCloses #2865 from andrewor14/active-executors and squashes the following commits:\n\nff3172b [Andrew Or] Add all known executors to `activeExecutorIds`","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/61ca7742d21dd66f5a7b3bb826e3aaca6f049b68"},{"sha":"1a623b2e163da3a9112cb9b68bda22b9e398ed5c","author":{"email":"23f994cfdead36bca960b16873361f50257dec1a@civisanalytics.com","name":"Michelangelo D'Agostino"},"message":"SPARK-3770: Make userFeatures accessible from python\n\nhttps://issues.apache.org/jira/browse/SPARK-3770\n\nWe need access to the underlying latent user features from python. However, the userFeatures RDD from the MatrixFactorizationModel isn't accessible from the python bindings. I've added a method to the underlying scala class to turn the RDD[(Int, Array[Double])] to an RDD[String]. This is then accessed from the python recommendation.py\n\nAuthor: Michelangelo D'Agostino <mdagostino@civisanalytics.com>\n\nCloses #2636 from mdagost/mf_user_features and squashes the following commits:\n\nc98f9e2 [Michelangelo D'Agostino] Added unit tests for userFeatures and productFeatures and merged master.\nd5eadf8 [Michelangelo D'Agostino] Merge branch 'master' into mf_user_features\n2481a2a [Michelangelo D'Agostino] Merged master and resolved conflict.\na6ffb96 [Michelangelo D'Agostino] Eliminated a function from our first approach to this problem that is no longer needed now that we added the fromTuple2RDD function.\n2aa1bf8 [Michelangelo D'Agostino] Implemented a function called fromTuple2RDD in PythonMLLibAPI and used it to expose the MF userFeatures and productFeatures in python.\n34cb2a2 [Michelangelo D'Agostino] A couple of lint cleanups and a comment.\ncdd98e3 [Michelangelo D'Agostino] It's working now.\ne1fbe5e [Michelangelo D'Agostino] Added scala function to stringify userFeatures for access in python.","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/1a623b2e163da3a9112cb9b68bda22b9e398ed5c"},{"sha":"5fdaf52a9df21cac69e2a4612aeb4e760e4424e7","author":{"email":"5a7a6a189b47cd31d24667a751e4003cd04eced5@databricks.com","name":"Aaron Davidson"},"message":"[SPARK-3994] Use standard Aggregator code path for countByKey and countByValue\n\nSee [JIRA](https://issues.apache.org/jira/browse/SPARK-3994) for more information. Also adds\na note which warns against using these methods.\n\nAuthor: Aaron Davidson <aaron@databricks.com>\n\nCloses #2839 from aarondav/countByKey and squashes the following commits:\n\nd6fdb2a [Aaron Davidson] Respond to comments\ne1f06d3 [Aaron Davidson] [SPARK-3994] Use standard Aggregator code path for countByKey and countByValue","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/5fdaf52a9df21cac69e2a4612aeb4e760e4424e7"},{"sha":"814a9cd7fabebf2a06f7e2e5d46b6a2b28b917c2","author":{"email":"cb19cdbb6f51a6c425c3962c36b8b3c147eed722@gmail.com","name":"coderxiang"},"message":"SPARK-3568 [mllib] add ranking metrics\n\nAdd common metrics for ranking algorithms (http://www-nlp.stanford.edu/IR-book/), including:\n - Mean Average Precision\n - Precisionn: top-n precision\n - Discounted cumulative gain (DCG) and NDCG\n\nThe following methods and the corresponding tests are implemented:\n\n```\nclass RankingMetrics[T](predictionAndLabels: RDD[(Array[T], Array[T])]) {\n  /* Returns the precsionk for each query */\n  lazy val precAtK: RDD[Array[Double]]\n\n  /**\n   * param k the position to compute the truncated precision\n   * return the average precision at the first k ranking positions\n   */\n  def precision(k: Int): Double\n\n  /* Returns the average precision for each query */\n  lazy val avePrec: RDD[Double]\n\n  /*Returns the mean average precision (MAP) of all the queries*/\n  lazy val meanAvePrec: Double\n\n  /*Returns the normalized discounted cumulative gain for each query */\n  lazy val ndcgAtK: RDD[Array[Double]]\n\n  /**\n   * param k the position to compute the truncated ndcg\n   * return the average ndcg at the first k ranking positions\n   */\n  def ndcg(k: Int): Double\n}\n```\n\nAuthor: coderxiang <shuoxiangpub@gmail.com>\n\nCloses #2667 from coderxiang/rankingmetrics and squashes the following commits:\n\nd881097 [coderxiang] update doc\n14d9cd9 [coderxiang] remove unexpected files\nd7fb93f [coderxiang] style change and remove ignored files\nf113ee1 [coderxiang] modify doc for displaying superscript and subscript\nf626896 [coderxiang] improve doc and remove unnecessary computation while labSet is empty\nbe6645e [coderxiang] set the precision of empty labset to 0.0\nd64c120 [coderxiang] add logWarning for empty ground truth set\ndfae292 [coderxiang] handle empty labSet for map. add test\n62047c4 [coderxiang] style change and add documentation\nf66612d [coderxiang] add additional test of precisionAt\nb794cb2 [coderxiang] move private members precAtK, ndcgAtK into public methods. style change\n77c9e5d [coderxiang] set precAtK and ndcgAtK as private member. Improve documentation\n5f87bce [coderxiang] add API to calculate precision and ndcg at each ranking position\nb7851cc [coderxiang] Use generic type to represent IDs\ne443fee [coderxiang] change style and use alternative builtin methods\n3a5a6ff [coderxiang] add ranking metrics","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/814a9cd7fabebf2a06f7e2e5d46b6a2b28b917c2"},{"sha":"856b081729057f9da31a86e4bfa0dc0013492042","author":{"email":"f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","name":"wangfei"},"message":"[SQL]redundant methods for broadcast\n\nredundant methods for broadcast in ```TableReader```\n\nAuthor: wangfei <wangfei1@huawei.com>\n\nCloses #2862 from scwf/TableReader and squashes the following commits:\n\n414cc24 [wangfei] unnecessary methods for broadcast","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/856b081729057f9da31a86e4bfa0dc0013492042"},{"sha":"6bb56faea8d238ea22c2de33db93b1b39f492b3a","author":{"email":"5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","name":"Sandy Ryza"},"message":"SPARK-1813. Add a utility to SparkConf that makes using Kryo really easy\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #789 from sryza/sandy-spark-1813 and squashes the following commits:\n\n48b05e9 [Sandy Ryza] Simplify\nb824932 [Sandy Ryza] Allow both spark.kryo.classesToRegister and spark.kryo.registrator at the same time\n6a15bb7 [Sandy Ryza] Small fix\na2278c0 [Sandy Ryza] Respond to review comments\n6ef592e [Sandy Ryza] SPARK-1813. Add a utility to SparkConf that makes using Kryo really easy","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/6bb56faea8d238ea22c2de33db93b1b39f492b3a"},{"sha":"bae4ca3bbf75cf5b57a098a95848b0fd65bc241e","author":{"email":"b88ad21539972f114bb4312435598d72d2e9e620@gmail.com","name":"Karthik"},"message":"Update JavaCustomReceiver.java\n\nChanged the usage string to correctly reflect the file name.\n\nAuthor: Karthik <karthik.gomadam@gmail.com>\n\nCloses #2699 from namelessnerd/patch-1 and squashes the following commits:\n\n8570e33 [Karthik] Update JavaCustomReceiver.java","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/bae4ca3bbf75cf5b57a098a95848b0fd65bc241e"},{"sha":"f05e09b4c95d799bdda3c3ff7fb76a4cd656415d","author":{"email":"fbb1b1249c4fec363b436c739458ff446a82c033@gmail.com","name":"CrazyJvm"},"message":"use isRunningLocally rather than runningLocally\n\nrunningLocally is deprecated now\n\nAuthor: CrazyJvm <crazyjvm@gmail.com>\n\nCloses #2879 from CrazyJvm/runningLocally and squashes the following commits:\n\nbec0b3e [CrazyJvm] use isRunningLocally rather than runningLocally","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/f05e09b4c95d799bdda3c3ff7fb76a4cd656415d"},{"sha":"97cf19f64e924569892e0a0417de19329855f4af","author":{"email":"05f4ff6cfd932210ddb781d81debbbf760fcab75@gmail.com","name":"freeman"},"message":"Fix for sampling error in NumPy v1.9 [SPARK-3995][PYSPARK]\n\nChange maximum value for default seed during RDD sampling so that it is strictly less than 2 ** 32. This prevents a bug in the most recent version of NumPy, which cannot accept random seeds above this bound.\n\nAdds an extra test that uses the default seed (instead of setting it manually, as in the docstrings).\n\nmengxr\n\nAuthor: freeman <the.freeman.lab@gmail.com>\n\nCloses #2889 from freeman-lab/pyspark-sampling and squashes the following commits:\n\ndc385ef [freeman] Change maximum value for default seed","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/97cf19f64e924569892e0a0417de19329855f4af"}]}