{"push_id":589490205,"size":5,"distinct_size":5,"ref":"refs/heads/master","head":"af2effdd7b54316af0c02e781911acfb148b962b","before":"39a54b40aff66816f8b8f5c6133eaaad6eaecae1","commits":[{"sha":"3efd8bb6cf139ce094ff631c7a9c1eb93fdcd566","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-6052][SQL]In JSON schema inference, we should always set containsNull of an ArrayType to true\n\nAlways set `containsNull = true` when infer the schema of JSON datasets. If we set `containsNull` based on records we scanned, we may miss arrays with null values when we do sampling. Also, because future data can have arrays with null values, if we convert JSON data to parquet, always setting `containsNull = true` is a more robust way to go.\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-6052\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #4806 from yhuai/jsonArrayContainsNull and squashes the following commits:\n\n05eab9d [Yin Huai] Change containsNull to true.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/3efd8bb6cf139ce094ff631c7a9c1eb93fdcd566"},{"sha":"95ac68bf127b5370c13d6bc15adbda78228829cc","author":{"email":"1fc3ade0ca66a5997ee9eee1a4074efe89a462b1@us.ibm.com","name":"Kenneth Myers"},"message":"[SPARK-6111] Fixed usage string in documentation.\n\nUsage info in documentation does not match actual usage info.\n\nDoc string usage says ```Usage: network_wordcount.py <zk> <topic>``` whereas the actual usage is ```Usage: kafka_wordcount.py <zk> <topic>```\n\nAuthor: Kenneth Myers <myerske@us.ibm.com>\n\nCloses #4852 from kennethmyers/kafka_wordcount_documentation_fix and squashes the following commits:\n\n3855325 [Kenneth Myers] Fixed usage string in documentation.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/95ac68bf127b5370c13d6bc15adbda78228829cc"},{"sha":"9ce12aaf283a2793e719bdc956dd858922636e8d","author":{"email":"62cbe0ac3409a518be51a2793bae027cd01385b1@huawei.com","name":"q00251598"},"message":"[SPARK-5741][SQL] Support the path contains comma in HiveContext\n\nWhen run ```select * from nzhang_part where hr = 'file,';```, it throws exception ```java.lang.IllegalArgumentException: Can not create a Path from an empty string```\n. Because the path of hdfs contains comma, and FileInputFormat.setInputPaths will split path by comma.\n\n### SQL\n```\nset hive.merge.mapfiles=true;\nset hive.merge.mapredfiles=true;\nset hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;\nset hive.exec.dynamic.partition=true;\nset hive.exec.dynamic.partition.mode=nonstrict;\n\ncreate table nzhang_part like srcpart;\n\ninsert overwrite table nzhang_part partition (ds='2010-08-15', hr) select key, value, hr from srcpart where ds='2008-04-08';\n\ninsert overwrite table nzhang_part partition (ds='2010-08-15', hr=11) select key, value from srcpart where ds='2008-04-08';\n\ninsert overwrite table nzhang_part partition (ds='2010-08-15', hr)\nselect * from (\nselect key, value, hr from srcpart where ds='2008-04-08'\nunion all\nselect '1' as key, '1' as value, 'file,' as hr from src limit 1) s;\n\nselect * from nzhang_part where hr = 'file,';\n```\n\n### Error Log\n```\n15/02/10 14:33:16 ERROR SparkSQLDriver: Failed in [select * from nzhang_part where hr = 'file,']\njava.lang.IllegalArgumentException: Can not create a Path from an empty string\nat org.apache.hadoop.fs.Path.checkPathArg(Path.java:127)\nat org.apache.hadoop.fs.Path.<init>(Path.java:135)\nat org.apache.hadoop.util.StringUtils.stringToPath(StringUtils.java:241)\nat org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:400)\nat org.apache.spark.sql.hive.HadoopTableReader$.initializeLocalJobConfFunc(TableReader.scala:251)\nat org.apache.spark.sql.hive.HadoopTableReader$$anonfun$11.apply(TableReader.scala:229)\nat org.apache.spark.sql.hive.HadoopTableReader$$anonfun$11.apply(TableReader.scala:229)\nat org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:172)\nat org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:172)\nat scala.Option.map(Option.scala:145)\nat org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:172)\nat org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:196)\n\nAuthor: q00251598 <qiyadong@huawei.com>\n\nCloses #4532 from watermen/SPARK-5741 and squashes the following commits:\n\n9758ab1 [q00251598] fix bug\n1db1a1c [q00251598] use setInputPaths(Job job, Path... inputPaths)\nb788a72 [q00251598] change FileInputFormat.setInputPaths to jobConf.set and add test suite","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/9ce12aaf283a2793e719bdc956dd858922636e8d"},{"sha":"e7d8ae444fead27fe85a879f2f7a4cfdd8c47b16","author":{"email":"a1724e1376b0a09607ae9806117b5aa49b7a4678@DEBORAHs-MacBook-Pro.local","name":"DEBORAH SIEGEL"},"message":"aggregateMessages example in graphX doc\n\nExamples illustrating difference between legacy mapReduceTriplets usage and aggregateMessages usage has type issues on the reduce for both operators.\n\nBeing just an example-  changed example to reduce the message String by concatenation. Although non-optimal for performance.\n\nAuthor: DEBORAH SIEGEL <deborahsiegel@DEBORAHs-MacBook-Pro.local>\n\nCloses #4853 from d3borah/master and squashes the following commits:\n\ndb54173 [DEBORAH SIEGEL] fixed aggregateMessages example in graphX doc","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/e7d8ae444fead27fe85a879f2f7a4cfdd8c47b16"},{"sha":"af2effdd7b54316af0c02e781911acfb148b962b","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-6080] [PySpark] correct LogisticRegressionWithLBFGS regType parameter for pyspark\n\nCurrently LogisticRegressionWithLBFGS in python/pyspark/mllib/classification.py will invoke callMLlibFunc with a wrong \"regType\" parameter.\nIt was assigned to \"str(regType)\" which translate None(Python) to \"None\"(Java/Scala). The right way should be translate None(Python) to null(Java/Scala) just as what we did at LogisticRegressionWithSGD.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #4831 from yanboliang/pyspark_classification and squashes the following commits:\n\n12db65a [Yanbo Liang] correct LogisticRegressionWithLBFGS regType parameter for pyspark","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/af2effdd7b54316af0c02e781911acfb148b962b"}]}