{"push_id":571065580,"size":44,"distinct_size":44,"ref":"refs/heads/SPARK-4879-sparkhadoopwriter-fix","head":"658116b57c1eeb785d9f3d4577b0a7631ffcfaf9","before":"14861eade5461214d3a2f2fd5f2f07738155c404","commits":[{"sha":"6fe70d8432314f0b7290a66f114306f61e0a87cc","author":{"email":"2e9bf58ec2f93170404b3a28538fb27400f699db@palantir.com","name":"mcheah"},"message":"[SPARK-5691] Fixing wrong data structure lookup for dupe app registratio...\n\nIn Master's registerApplication method, it checks if the application had\nalready registered by examining the addressToWorker hash map. In reality,\nit should refer to the addressToApp data structure, as this is what\nreally tracks which apps have been registered.\n\nAuthor: mcheah <mcheah@palantir.com>\n\nCloses #4477 from mccheah/spark-5691 and squashes the following commits:\n\nefdc573 [mcheah] [SPARK-5691] Fixing wrong data structure lookup for dupe app registration","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/6fe70d8432314f0b7290a66f114306f61e0a87cc"},{"sha":"0765af9b21e9204c410c7a849c7201bc3eda8cc3","author":{"email":"e1f8d9790f67825b23faf6a373ab7c83bc8900f2@apache.org","name":"Hari Shreedharan"},"message":"[SPARK-4905][STREAMING] FlumeStreamSuite fix.\n\nUsing String constructor instead of CharsetDecoder to see if it fixes the issue of empty strings in Flume test output.\n\nAuthor: Hari Shreedharan <hshreedharan@apache.org>\n\nCloses #4371 from harishreedharan/Flume-stream-attempted-fix and squashes the following commits:\n\n550d363 [Hari Shreedharan] Fix imports.\n8695950 [Hari Shreedharan] Use Charsets.UTF_8 instead of \"UTF-8\" in String constructors.\naf3ba14 [Hari Shreedharan] [SPARK-4905][STREAMING] FlumeStreamSuite fix.","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/0765af9b21e9204c410c7a849c7201bc3eda8cc3"},{"sha":"f48199eb354d6ec8675c2c1f96c3005064058d66","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-5675][SQL] XyzType companion object should subclass XyzType\n\nOtherwise, the following will always return false in Java.\n\n```scala\ndataType instanceof StringType\n```\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #4463 from rxin/type-companion-object and squashes the following commits:\n\n04d5d8d [Reynold Xin] Comment.\n976e11e [Reynold Xin] [SPARK-5675][SQL]StringType case object should be subclass of StringType class","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/f48199eb354d6ec8675c2c1f96c3005064058d66"},{"sha":"b884daa58084d4f42e2318894067565b94e07f9d","author":{"email":"402011891a3cdeb0b47054035df68045f94f3c49@gmail.com","name":"Florian Verhein"},"message":"[SPARK-5611] [EC2] Allow spark-ec2 repo and branch to be set on CLI of spark_ec2.py\n\nand by extension, the ami-list\n\nUseful for using alternate spark-ec2 repos or branches.\n\nAuthor: Florian Verhein <florian.verhein@gmail.com>\n\nCloses #4385 from florianverhein/master and squashes the following commits:\n\n7e2b4be [Florian Verhein] [SPARK-5611] [EC2] typo\n8b653dc [Florian Verhein] [SPARK-5611] [EC2] Enforce only supporting spark-ec2 forks from github, log improvement\nbc4b0ed [Florian Verhein] [SPARK-5611] allow spark-ec2 repos with different names\n8b5c551 [Florian Verhein] improve option naming, fix logging, fix lint failing, add guard to enforce spark-ec2\n7724308 [Florian Verhein] [SPARK-5611] [EC2] fixes\nb42b68c [Florian Verhein] [SPARK-5611] [EC2] Allow spark-ec2 repo and branch to be set on CLI of spark_ec2.py","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/b884daa58084d4f42e2318894067565b94e07f9d"},{"sha":"68b25cf695e0fce9e465288d5a053e540a3fccb4","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SQL] Add some missing DataFrame functions.\n\n- as with a `Symbol`\n- distinct\n- sqlContext.emptyDataFrame\n- move add/remove col out of RDDApi section\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #4437 from marmbrus/dfMissingFuncs and squashes the following commits:\n\n2004023 [Michael Armbrust] Add missing functions","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/68b25cf695e0fce9e465288d5a053e540a3fccb4"},{"sha":"5f0b30e59cc6a3017168189d3aaf09402699dc3b","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SQL] Code cleanup.\n\nI added an unnecessary line of code in https://github.com/apache/spark/commit/13531dd97c08563e53dacdaeaf1102bdd13ef825.\n\nMy bad. Let's delete it.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #4482 from yhuai/unnecessaryCode and squashes the following commits:\n\n3645af0 [Yin Huai] Code cleanup.","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/5f0b30e59cc6a3017168189d3aaf09402699dc3b"},{"sha":"b8080aa86d55e0467fd4328f10a2f0d6605e6cc6","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-5696] [SQL] [HOTFIX] Asks HiveThriftServer2 to re-initialize log4j using Hive configurations\n\nIn this way, log4j configurations overriden by jets3t-0.9.2.jar can be again overriden by Hive default log4j configurations.\n\nThis might not be the best solution for this issue since it requires users to use `hive-log4j.properties` rather than `log4j.properties` to initialize `HiveThriftServer2` logging configurations, which can be confusing. The main purpose of this PR is to fix Jenkins PR build.\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/apache/spark/4484)\n<!-- Reviewable:end -->\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #4484 from liancheng/spark-5696 and squashes the following commits:\n\ndf83956 [Cheng Lian] Hot fix: asks HiveThriftServer2 to re-initialize log4j using Hive configurations","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/b8080aa86d55e0467fd4328f10a2f0d6605e6cc6"},{"sha":"2a36292534a1e9f7a501e88f69bfc3a09fb62cb3","author":{"email":"1d4686049270c5b7761703671390519558ef1f37@baidu.com","name":"Lu Yan"},"message":"[SPARK-5614][SQL] Predicate pushdown through Generate.\n\nNow in Catalyst's rules, predicates can not be pushed through \"Generate\" nodes. Further more, partition pruning in HiveTableScan can not be applied on those queries involves \"Generate\". This makes such queries very inefficient. In practice, it finds patterns like\n\n```scala\nFilter(predicate, Generate(generator, _, _, _, grandChild))\n```\n\nand splits the predicate into 2 parts by referencing the generated column from Generate node or not. And a new Filter will be created for those conjuncts can be pushed beneath Generate node. If nothing left for the original Filter, it will be removed.\nFor example, physical plan for query\n```sql\nselect len, bk\nfrom s_server lateral view explode(len_arr) len_table as len\nwhere len > 5 and day = '20150102';\n```\nwhere 'day' is a partition column in metastore\u0010 is like this in current version of Spark SQL:\n\n> Project [len, bk]\n>\n> Filter ((len > \"5\") && \"(day = \"20150102\")\")\n>\n> Generate explode(len_arr), true, false\n>\n> HiveTableScan [bk, len_arr, day], (MetastoreRelation default, s_server, None), None\n\nBut theoretically the plan should be like this\n\n> Project [len, bk]\n>\n> Filter (len > \"5\")\n>\n> Generate explode(len_arr), true, false\n>\n> HiveTableScan [bk, len_arr, day], (MetastoreRelation default, s_server, None), Some(day = \"20150102\")\n\nWhere partition pruning predicates can be pushed to HiveTableScan nodes.\n\nAuthor: Lu Yan <luyan02@baidu.com>\n\nCloses #4394 from ianluyan/ppd and squashes the following commits:\n\na67dce9 [Lu Yan] Fix English grammar.\n7cea911 [Lu Yan] Revised based on @marmbrus's opinions\nffc59fc [Lu Yan] [SPARK-5614][SQL] Predicate pushdown through Generate.","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/2a36292534a1e9f7a501e88f69bfc3a09fb62cb3"},{"sha":"0ee53ebce9944722e76b2b28fae79d9956be9f17","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@outlook.com","name":"Wenchen Fan"},"message":"[SPARK-2096][SQL] support dot notation on array of struct\n\n~~The rule is simple: If you want `a.b` work, then `a` must be some level of nested array of struct(level 0 means just a StructType). And the result of `a.b` is same level of nested array of b-type.\nAn optimization is: the resolve chain looks like `Attribute -> GetItem -> GetField -> GetField ...`, so we could transmit the nested array information between `GetItem` and `GetField` to avoid repeated computation of `innerDataType` and `containsNullList` of that nested array.~~\nmarmbrus Could you take a look?\n\nto evaluate `a.b`, if `a` is array of struct, then `a.b` means get field `b` on each element of `a`, and return a result of array.\n\nAuthor: Wenchen Fan <cloud0fan@outlook.com>\n\nCloses #2405 from cloud-fan/nested-array-dot and squashes the following commits:\n\n08a228a [Wenchen Fan] support dot notation on array of struct","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/0ee53ebce9944722e76b2b28fae79d9956be9f17"},{"sha":"d08e7c2b498584609cb3c7922eaaa2a0d115603f","author":{"email":"7acf0e49bc9acc50da7e38d41884e81d5f2df305@qq.com","name":"DoingDone9"},"message":"[SPARK-5648][SQL] support \"alter ... unset tblproperties(\"key\")\"\n\nmake hivecontext support \"alter ... unset tblproperties(\"key\")\"\nlike :\nalter view viewName unset tblproperties(\"k\")\nalter table tableName unset tblproperties(\"k\")\n\nAuthor: DoingDone9 <799203320@qq.com>\n\nCloses #4424 from DoingDone9/unset and squashes the following commits:\n\n6dd8bee [DoingDone9] support \"alter ... unset tblproperties(\"key\")\"","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/d08e7c2b498584609cb3c7922eaaa2a0d115603f"},{"sha":"3ec3ad295ddd1435da68251b7479ffb60aec7037","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-5699] [SQL] [Tests] Runs hive-thriftserver tests whenever SQL code is modified\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/apache/spark/4486)\n<!-- Reviewable:end -->\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #4486 from liancheng/spark-5699 and squashes the following commits:\n\n538001d [Cheng Lian] Runs hive-thriftserver tests whenever SQL code is modified","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/3ec3ad295ddd1435da68251b7479ffb60aec7037"},{"sha":"d302c4800bf2f74eceb731169ddf1766136b7398","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-5698] Do not let user request negative # of executors\n\nOtherwise we might crash the ApplicationMaster. Why? Please see https://issues.apache.org/jira/browse/SPARK-5698.\n\nsryza I believe this is also relevant in your patch #4168.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #4483 from andrewor14/da-negative and squashes the following commits:\n\n53ed955 [Andrew Or] Throw IllegalArgumentException instead\n0e89fd5 [Andrew Or] Check against negative requests","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/d302c4800bf2f74eceb731169ddf1766136b7398"},{"sha":"08488c175f2e8532cb6aab84da2abd9ad57179cc","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-5469] restructure pyspark.sql into multiple files\n\nAll the DataTypes moved into pyspark.sql.types\n\nThe changes can be tracked by `--find-copies-harder -M25`\n```\ndavieslocalhost:~/work/spark/python$ git diff --find-copies-harder -M25 --numstat master..\n2       5       python/docs/pyspark.ml.rst\n0       3       python/docs/pyspark.mllib.rst\n10      2       python/docs/pyspark.sql.rst\n1       1       python/pyspark/mllib/linalg.py\n21      14      python/pyspark/{mllib => sql}/__init__.py\n14      2108    python/pyspark/{sql.py => sql/context.py}\n10      1772    python/pyspark/{sql.py => sql/dataframe.py}\n7       6       python/pyspark/{sql_tests.py => sql/tests.py}\n8       1465    python/pyspark/{sql.py => sql/types.py}\n4       2       python/run-tests\n1       1       sql/core/src/main/scala/org/apache/spark/sql/test/ExamplePointUDT.scala\n```\n\nAlso `git blame -C -C python/pyspark/sql/context.py` to track the history.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #4479 from davies/sql and squashes the following commits:\n\n1b5f0a5 [Davies Liu] Merge branch 'master' of github.com:apache/spark into sql\n2b2b983 [Davies Liu] restructure pyspark.sql","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/08488c175f2e8532cb6aab84da2abd9ad57179cc"},{"sha":"31d435ecfdc24a788a6e38f4e82767bc275a3283","author":{"email":"1bbfbf3ad5c4de88db64d0be3ca037b409d25231@huawei.com","name":"KaiXinXiaoLei"},"message":"Add a config option to print DAG.\n\nAdd a config option \"spark.rddDebug.enable\" to check whether to print DAG info. When \"spark.rddDebug.enable\" is true, it will print information about DAG in the log.\n\nAuthor: KaiXinXiaoLei <huleilei1@huawei.com>\n\nCloses #4257 from KaiXinXiaoLei/DAGprint and squashes the following commits:\n\nd9fe42e [KaiXinXiaoLei] change  log info\nc27ee76 [KaiXinXiaoLei] change log info\n83c2b32 [KaiXinXiaoLei] change config option\nadcb14f [KaiXinXiaoLei] change the file.\nf4e7b9e [KaiXinXiaoLei] add a option to print DAG","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/31d435ecfdc24a788a6e38f4e82767bc275a3283"},{"sha":"36c4e1d75933dc843acb747b91dc12e75ad1df42","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"SPARK-4900 [MLLIB] MLlib SingularValueDecomposition ARPACK IllegalStateException\n\nFix ARPACK error code mapping, at least. It's not yet clear whether the error is what we expect from ARPACK. If it isn't, not sure if that's to be treated as an MLlib or Breeze issue.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #4485 from srowen/SPARK-4900 and squashes the following commits:\n\n7355aa1 [Sean Owen] Fix ARPACK error code mapping","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/36c4e1d75933dc843acb747b91dc12e75ad1df42"},{"sha":"20a6013106b56a1a1cc3e8cda092330ffbe77cc3","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-2996] Implement userClassPathFirst for driver, yarn.\n\nYarn's config option `spark.yarn.user.classpath.first` does not work the same way as\n`spark.files.userClassPathFirst`; Yarn's version is a lot more dangerous, in that it\nmodifies the system classpath, instead of restricting the changes to the user's class\nloader. So this change implements the behavior of the latter for Yarn, and deprecates\nthe more dangerous choice.\n\nTo be able to achieve feature-parity, I also implemented the option for drivers (the existing\noption only applies to executors). So now there are two options, each controlling whether\nto apply userClassPathFirst to the driver or executors. The old option was deprecated, and\naliased to the new one (`spark.executor.userClassPathFirst`).\n\nThe existing \"child-first\" class loader also had to be fixed. It didn't handle resources, and it\nwas also doing some things that ended up causing JVM errors depending on how things\nwere being called.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #3233 from vanzin/SPARK-2996 and squashes the following commits:\n\n9cf9cf1 [Marcelo Vanzin] Merge branch 'master' into SPARK-2996\na1499e2 [Marcelo Vanzin] Remove SPARK_HOME propagation.\nfa7df88 [Marcelo Vanzin] Remove 'test.resource' file, create it dynamically.\na8c69f1 [Marcelo Vanzin] Review feedback.\ncabf962 [Marcelo Vanzin] Merge branch 'master' into SPARK-2996\na1b8d7e [Marcelo Vanzin] Merge branch 'master' into SPARK-2996\n3f768e3 [Marcelo Vanzin] Merge branch 'master' into SPARK-2996\n2ce3c7a [Marcelo Vanzin] Merge branch 'master' into SPARK-2996\n0e6d6be [Marcelo Vanzin] Merge branch 'master' into SPARK-2996\n70d4044 [Marcelo Vanzin] Fix pyspark/yarn-cluster test.\n0fe7777 [Marcelo Vanzin] Merge branch 'master' into SPARK-2996\n0e6ef19 [Marcelo Vanzin] Move class loaders around and make names more meaninful.\nfe970a7 [Marcelo Vanzin] Review feedback.\n25d4fed [Marcelo Vanzin] Merge branch 'master' into SPARK-2996\n3cb6498 [Marcelo Vanzin] Call the right loadClass() method on the parent.\nfbb8ab5 [Marcelo Vanzin] Add locking in loadClass() to avoid deadlocks.\n2e6c4b7 [Marcelo Vanzin] Mention new setting in documentation.\nb6497f9 [Marcelo Vanzin] Merge branch 'master' into SPARK-2996\na10f379 [Marcelo Vanzin] Some feedback.\n3730151 [Marcelo Vanzin] Merge branch 'master' into SPARK-2996\nf513871 [Marcelo Vanzin] Merge branch 'master' into SPARK-2996\n44010b6 [Marcelo Vanzin] Merge branch 'master' into SPARK-2996\n7b57cba [Marcelo Vanzin] Remove now outdated message.\n5304d64 [Marcelo Vanzin] Merge branch 'master' into SPARK-2996\n35949c8 [Marcelo Vanzin] Merge branch 'master' into SPARK-2996\n54e1a98 [Marcelo Vanzin] Merge branch 'master' into SPARK-2996\nd1273b2 [Marcelo Vanzin] Add test file to rat exclude.\nfa1aafa [Marcelo Vanzin] Remove write check on user jars.\n89d8072 [Marcelo Vanzin] Cleanups.\na963ea3 [Marcelo Vanzin] Implement spark.driver.userClassPathFirst for standalone cluster mode.\n50afa5f [Marcelo Vanzin] Fix Yarn executor command line.\n7d14397 [Marcelo Vanzin] Register user jars in executor up front.\n7f8603c [Marcelo Vanzin] Fix yarn-cluster mode without userClassPathFirst.\n20373f5 [Marcelo Vanzin] Fix ClientBaseSuite.\n55c88fa [Marcelo Vanzin] Run all Yarn integration tests via spark-submit.\n0b64d92 [Marcelo Vanzin] Add deprecation warning to yarn option.\n4a84d87 [Marcelo Vanzin] Fix the child-first class loader.\nd0394b8 [Marcelo Vanzin] Add \"deprecated configs\" to SparkConf.\n46d8cf2 [Marcelo Vanzin] Update doc with new option, change name to \"userClassPathFirst\".\na314f2d [Marcelo Vanzin] Enable driver class path isolation in SparkSubmit.\n91f7e54 [Marcelo Vanzin] [yarn] Enable executor class path isolation.\na853e74 [Marcelo Vanzin] Re-work CoarseGrainedExecutorBackend command line arguments.\n89522ef [Marcelo Vanzin] Add class path isolation support for Yarn cluster mode.","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/20a6013106b56a1a1cc3e8cda092330ffbe77cc3"},{"sha":"a95ed52157473fb0e42e910ee15270e7f0edf943","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-5703] AllJobsPage throws empty.max exception\n\nIf you have a `SparkListenerJobEnd` event without the corresponding `SparkListenerJobStart` event, then `JobProgressListener` will create an empty `JobUIData` with an empty `stageIds` list. However, later in `AllJobsPage` we call `stageIds.max`. If this is empty, it will throw an exception.\n\nThis crashed my history server.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #4490 from andrewor14/jobs-page-max and squashes the following commits:\n\n21797d3 [Andrew Or] Check nonEmpty before calling max","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/a95ed52157473fb0e42e910ee15270e7f0edf943"},{"sha":"a2d33d0b01af87e931d9d883638a52d7a86f6248","author":{"email":"284a00ea9901227c42561196ff8576aed3ac5631@gmail.com","name":"Kay Ousterhout"},"message":"[SPARK-5701] Only set ShuffleReadMetrics when task has shuffle deps\n\nThe updateShuffleReadMetrics method in TaskMetrics (called by the executor heartbeater) will currently always add a ShuffleReadMetrics to TaskMetrics (with values set to 0), even when the task didn't read any shuffle data. ShuffleReadMetrics should only be added if the task reads shuffle data.\n\nAuthor: Kay Ousterhout <kayousterhout@gmail.com>\n\nCloses #4488 from kayousterhout/SPARK-5701 and squashes the following commits:\n\n673ed58 [Kay Ousterhout] SPARK-5701: Only set ShuffleReadMetrics when task has shuffle deps","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/a2d33d0b01af87e931d9d883638a52d7a86f6248"},{"sha":"bd0b5ea708aa5b84adb67c039ec52408289718bb","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SQL] Remove the duplicated code\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #4494 from chenghao-intel/tiny_code_change and squashes the following commits:\n\n450dfe7 [Cheng Hao] remove the duplicated code","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/bd0b5ea708aa5b84adb67c039ec52408289718bb"},{"sha":"ef2f55b97f58fa06acb30e9e0172fb66fba383bc","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-5597][MLLIB] save/load for decision trees and emsembles\n\nThis is based on #4444 from jkbradley with the following changes:\n\n1. Node schema updated to\n   ~~~\ntreeId: int\nnodeId: Int\npredict/\n       |- predict: Double\n       |- prob: Double\nimpurity: Double\nisLeaf: Boolean\nsplit/\n     |- feature: Int\n     |- threshold: Double\n     |- featureType: Int\n     |- categories: Array[Double]\nleftNodeId: Integer\nrightNodeId: Integer\ninfoGain: Double\n~~~\n\n2. Some refactor of the implementation.\n\nCloses #4444.\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #4493 from mengxr/SPARK-5597 and squashes the following commits:\n\n75e3bb6 [Xiangrui Meng] fix style\n2b0033d [Xiangrui Meng] update tree export schema and refactor the implementation\n45873a2 [Joseph K. Bradley] org imports\n1d4c264 [Joseph K. Bradley] Added save/load for tree ensembles\ndcdbf85 [Joseph K. Bradley] added save/load for decision tree but need to generalize it to ensembles","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/ef2f55b97f58fa06acb30e9e0172fb66fba383bc"}]}