{"push_id":571112660,"size":194,"distinct_size":138,"ref":"refs/heads/schema","head":"535b0b6896bae2c0540ba29991a3f083545d52b2","before":"d6c93c54eadfc216e29377920f2c4937ae73cd4e","commits":[{"sha":"0561c4544967fb853419f32e014fac9b8879b0db","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-5154] [PySpark] [Streaming] Kafka streaming support in Python\n\nThis PR brings the Python API for Spark Streaming Kafka data source.\n\n```\n    class KafkaUtils(__builtin__.object)\n     |  Static methods defined here:\n     |\n     |  createStream(ssc, zkQuorum, groupId, topics, storageLevel=StorageLevel(True, True, False, False,\n2), keyDecoder=<function utf8_decoder>, valueDecoder=<function utf8_decoder>)\n     |      Create an input stream that pulls messages from a Kafka Broker.\n     |\n     |      :param ssc:  StreamingContext object\n     |      :param zkQuorum:  Zookeeper quorum (hostname:port,hostname:port,..).\n     |      :param groupId:  The group id for this consumer.\n     |      :param topics:  Dict of (topic_name -> numPartitions) to consume.\n     |                      Each partition is consumed in its own thread.\n     |      :param storageLevel:  RDD storage level.\n     |      :param keyDecoder:  A function used to decode key\n     |      :param valueDecoder:  A function used to decode value\n     |      :return: A DStream object\n```\nrun the example:\n\n```\nbin/spark-submit --driver-class-path external/kafka-assembly/target/scala-*/spark-streaming-kafka-assembly-*.jar examples/src/main/python/streaming/kafka_wordcount.py localhost:2181 test\n```\n\nAuthor: Davies Liu <davies@databricks.com>\nAuthor: Tathagata Das <tdas@databricks.com>\n\nCloses #3715 from davies/kafka and squashes the following commits:\n\nd93bfe0 [Davies Liu] Update make-distribution.sh\n4280d04 [Davies Liu] address comments\ne6d0427 [Davies Liu] Merge branch 'master' of github.com:apache/spark into kafka\nf257071 [Davies Liu] add tests for null in RDD\n23b039a [Davies Liu] address comments\n9af51c4 [Davies Liu] Merge branch 'kafka' of github.com:davies/spark into kafka\na74da87 [Davies Liu] address comments\ndc1eed0 [Davies Liu] Update kafka_wordcount.py\n31e2317 [Davies Liu] Update kafka_wordcount.py\n370ba61 [Davies Liu] Update kafka.py\n97386b3 [Davies Liu] address comment\n2c567a5 [Davies Liu] update logging and comment\n33730d1 [Davies Liu] Merge branch 'master' of github.com:apache/spark into kafka\nadeeb38 [Davies Liu] Merge pull request #3 from tdas/kafka-python-api\naea8953 [Tathagata Das] Kafka-assembly for Python API\neea16a7 [Davies Liu] refactor\nf6ce899 [Davies Liu] add example and fix bugs\n98c8d17 [Davies Liu] fix python style\n5697a01 [Davies Liu] bypass decoder in scala\n048dbe6 [Davies Liu] fix python style\n75d485e [Davies Liu] add mqtt\n07923c4 [Davies Liu] support kafka in Python","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/0561c4544967fb853419f32e014fac9b8879b0db"},{"sha":"1bcd46574e442e20f55709d70573f271ce44e5b9","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-5512][Mllib] Run the PIC algorithm with initial vector suggected by the PIC paper\n\nAs suggested by the paper of Power Iteration Clustering, it is useful to set the initial vector v0 as the degree vector d. This pr tries to add a running method for that.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #4301 from viirya/pic_degreevector and squashes the following commits:\n\n7db28fb [Liang-Chi Hsieh] Refactor it to address comments.\n19cf94e [Liang-Chi Hsieh] Add an option to select initialization method.\nec88567 [Liang-Chi Hsieh] Run the PIC algorithm with degree vector d as suggected by the PIC paper.","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/1bcd46574e442e20f55709d70573f271ce44e5b9"},{"sha":"8f471a66db0571a76a21c0d93312197fee16174a","author":{"email":"004048764b4bb6d8761a43e498bc48ea4c4ed5a1@gmail.com","name":"Tor Myklebust"},"message":"[SPARK-5472][SQL] A JDBC data source for Spark SQL.\n\nThis pull request contains a Spark SQL data source that can pull data from, and can put data into, a JDBC database.\n\nI have tested both read and write support with H2, MySQL, and Postgres.  It would surprise me if both read and write support worked flawlessly out-of-the-box for any other database; different databases have different names for different JDBC data types and different meanings for SQL types with the same name.  However, this code is designed (see `DriverQuirks.scala`) to make it *relatively* painless to add support for another database by augmenting the type mapping contained in this PR.\n\nAuthor: Tor Myklebust <tmyklebu@gmail.com>\n\nCloses #4261 from tmyklebu/master and squashes the following commits:\n\ncf167ce [Tor Myklebust] Work around other Java tests ruining TestSQLContext.\n67893bf [Tor Myklebust] Move the jdbcRDD methods into SQLContext itself.\n585f95b [Tor Myklebust] Dependencies go into the project's pom.xml.\n829d5ba [Tor Myklebust] Merge branch 'master' of https://github.com/apache/spark\n41647ef [Tor Myklebust] Hide a couple things that don't need to be public.\n7318aea [Tor Myklebust] Fix scalastyle warnings.\na09eeac [Tor Myklebust] JDBC data source for Spark SQL.\n176bb98 [Tor Myklebust] Add test deps for JDBC support.","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/8f471a66db0571a76a21c0d93312197fee16174a"},{"sha":"cb39f120430fe44695f38d80858c9aed79eedd6f","author":{"email":"2e8aa918660411855c6d44d5bb2da677aa033255@nemccarthy.me","name":"nemccarthy"},"message":"[SPARK-5543][WebUI] Remove unused import JsonUtil from from JsonProtocol\n\nSimple PR to Remove unused import JsonUtil from from org.apache.spark.util.JsonProtocol.scala which fails builds with older versions of hadoop-core\nThis import is unused. It was introduced in PR #4029 https://github.com/apache/spark/pull/4029 as a part of JIRA SPARK-5231\n\nAuthor: nemccarthy <nathan@nemccarthy.me>\n\nCloses #4320 from nemccarthy/master and squashes the following commits:\n\n8e34a11 [nemccarthy] [SPARK-5543][WebUI] Remove unused import JsonUtil from from org.apache.spark.util.JsonProtocol.scala which fails builds with older versions of hadoop-core","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/cb39f120430fe44695f38d80858c9aed79eedd6f"},{"sha":"0ef38f5fad637fe96464778c1e5ec99cfcc6313c","author":{"email":"cbb7353e6d953ef360baf960c122346276c6e320@databricks.com","name":"Patrick Wendell"},"message":"SPARK-5542: Decouple publishing, packaging, and tagging in release script\n\nThese are some changes to the build script to allow parts of it to be run independently. This has already been tested during the 1.2.1 release cycle.\n\nAuthor: Patrick Wendell <patrick@databricks.com>\nAuthor: Patrick Wendell <pwendell@gmail.com>\n\nCloses #4319 from pwendell/release-updates and squashes the following commits:\n\ndfe7ed9 [Patrick Wendell] Merge remote-tracking branch 'apache/master' into release-updates\n478b072 [Patrick Wendell] Merge remote-tracking branch 'apache/master' into release-updates\n126dd0c [Patrick Wendell] Allow decoupling Maven publishing from cutting release","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/0ef38f5fad637fe96464778c1e5ec99cfcc6313c"},{"sha":"7930d2bef0e2c7f62456e013124455061dfe6dc8","author":{"email":"cbb7353e6d953ef360baf960c122346276c6e320@databricks.com","name":"Patrick Wendell"},"message":"SPARK-3996: Add jetty servlet and continuations.\n\nThese are needed transitively from the other Jetty libraries\nwe include. It was not picked up by unit tests because we\ndisable the UI.\n\nAuthor: Patrick Wendell <patrick@databricks.com>\n\nCloses #4323 from pwendell/jetty and squashes the following commits:\n\nd8669da [Patrick Wendell] SPARK-3996: Add jetty servlet and continuations.","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/7930d2bef0e2c7f62456e013124455061dfe6dc8"},{"sha":"60f67e7a142f831f91f60676f94affa8add9944f","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[Doc] Minor: Fixes several formatting issues\n\nFixes several minor formatting issues in the [Continuous Compilation] [1] section.\n\n[1]: http://spark.apache.org/docs/latest/building-spark.html#continuous-compilation\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/apache/spark/4316)\n<!-- Reviewable:end -->\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #4316 from liancheng/fix-build-instruction-docs and squashes the following commits:\n\n0a92e01 [Cheng Lian] Fixes several formatting issues","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/60f67e7a142f831f91f60676f94affa8add9944f"},{"sha":"c306555f491e45ef870f58938af397f9ec5f166a","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-5219][Core] Add locks to avoid scheduling race conditions\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #4019 from zsxwing/SPARK-5219 and squashes the following commits:\n\n36a8b4e [zsxwing] Add locks to avoid race conditions","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/c306555f491e45ef870f58938af397f9ec5f166a"},{"sha":"eb0da6c4bd55aaab972c53eb934e68257b8994e5","author":{"email":"05f4ff6cfd932210ddb781d81debbbf760fcab75@gmail.com","name":"freeman"},"message":"[SPARK-4979][MLLIB] Streaming logisitic regression\n\nThis adds support for streaming logistic regression with stochastic gradient descent, in the same manner as the existing implementation of streaming linear regression. It is a relatively simple addition because most of the work is already done by the abstract class `StreamingLinearAlgorithm` and existing algorithms and models from MLlib.\n\nThe PR includes\n- Streaming Logistic Regression algorithm\n- Unit tests for accuracy, streaming convergence, and streaming prediction\n- An example use\n\ncc mengxr tdas\n\nAuthor: freeman <the.freeman.lab@gmail.com>\n\nCloses #4306 from freeman-lab/streaming-logisitic-regression and squashes the following commits:\n\n5c2c70b [freeman] Use Option on model\n5cca2bc [freeman] Merge remote-tracking branch 'upstream/master' into streaming-logisitic-regression\n275f8bd [freeman] Make private to mllib\n3926e4e [freeman] Line formatting\n5ee8694 [freeman] Experimental tag for docs\n2fc68ac [freeman] Fix example formatting\n85320b1 [freeman] Fixed line length\nd88f717 [freeman] Remove stray comment\n59d7ecb [freeman] Add streaming logistic regression\ne78fe28 [freeman] Add streaming logistic regression example\n321cc66 [freeman] Set private and protected within mllib","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/eb0da6c4bd55aaab972c53eb934e68257b8994e5"},{"sha":"c31c36c4a76bd3449696383321332ec95bff7fed","author":{"email":"a1a405b82800b492a7268cba6bc58270d2b527de@apache.org","name":"Thomas Graves"},"message":"[SPARK-3778] newAPIHadoopRDD doesn't properly pass credentials for secure hdfs\n\n.this was https://github.com/apache/spark/pull/2676\n\nhttps://issues.apache.org/jira/browse/SPARK-3778\n\nThis affects if someone is trying to access secure hdfs something like:\nval lines = {\nval hconf = new Configuration()\nhconf.set(\"mapred.input.dir\", \"mydir\")\nhconf.set(\"textinputformat.record.delimiter\",\"\\003432\\n\")\nsc.newAPIHadoopRDD(hconf, classOf[TextInputFormat], classOf[LongWritable], classOf[Text])\n}\n\nAuthor: Thomas Graves <tgraves@apache.org>\n\nCloses #4292 from tgravescs/SPARK-3788 and squashes the following commits:\n\ncf3b453 [Thomas Graves] newAPIHadoopRDD doesn't properly pass credentials for secure hdfs on yarn","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/c31c36c4a76bd3449696383321332ec95bff7fed"},{"sha":"50a1a874e1d087a6c79835b1936d0009622a97b1","author":{"email":"2968329f6725f5640dc075ac70ec6ce5508c7fbc@flytxt.com","name":"FlytxtRnD"},"message":"[SPARK-5012][MLLib][PySpark]Python API for Gaussian Mixture Model\n\nPython API for the Gaussian Mixture Model clustering algorithm in MLLib.\n\nAuthor: FlytxtRnD <meethu.mathew@flytxt.com>\n\nCloses #4059 from FlytxtRnD/PythonGmmWrapper and squashes the following commits:\n\nc973ab3 [FlytxtRnD] Merge branch 'PythonGmmWrapper', remote-tracking branch 'upstream/master' into PythonGmmWrapper\n339b09c [FlytxtRnD] Added MultivariateGaussian namedtuple  and Arraybuffer in trainGaussianMixture\nfa0a142 [FlytxtRnD] New line added\nd5b36ab [FlytxtRnD] Changed argument names to lowercase\nac134f1 [FlytxtRnD] Merge branch 'PythonGmmWrapper' of https://github.com/FlytxtRnD/spark into PythonGmmWrapper\n6671ea1 [FlytxtRnD] Added mllib/stat/distribution.py\n3aee84b [FlytxtRnD] Fixed style issues\n2e9f12a [FlytxtRnD] Added mllib/stat/distribution.py and fixed style issues\nb22532c [FlytxtRnD] Merge branch 'PythonGmmWrapper', remote-tracking branch 'upstream/master' into PythonGmmWrapper\n2e14d82 [FlytxtRnD] Incorporate MultivariateGaussian instances in GaussianMixtureModel\n05767c7 [FlytxtRnD] Merge branch 'PythonGmmWrapper', remote-tracking branch 'upstream/master' into PythonGmmWrapper\n3464d19 [FlytxtRnD] Merge branch 'PythonGmmWrapper', remote-tracking branch 'upstream/master' into PythonGmmWrapper\nc1d4c71 [FlytxtRnD] Merge branch 'PythonGmmWrapper', remote-tracking branch 'origin/PythonGmmWrapper' into PythonGmmWrapper\n426d130 [FlytxtRnD] Added random seed parameter\n332bad1 [FlytxtRnD] Merge branch 'PythonGmmWrapper', remote-tracking branch 'upstream/master' into PythonGmmWrapper\nf82750b [FlytxtRnD] Fixed style issues\n5c83825 [FlytxtRnD] Split input file with space delimiter\nfda60f3 [FlytxtRnD] Python API for Gaussian Mixture Model","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/50a1a874e1d087a6c79835b1936d0009622a97b1"},{"sha":"13531dd97c08563e53dacdaeaf1102bdd13ef825","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-5501][SPARK-5420][SQL] Write support for the data source API\n\nThis PR aims to support `INSERT INTO/OVERWRITE TABLE tableName` and `CREATE TABLE tableName AS SELECT` for the data source API (partitioned tables are not supported).\n\nIn this PR, I am also adding the support of `IF NOT EXISTS` for our ddl parser. The current semantic of `IF NOT EXISTS` is explained as follows.\n* For a `CREATE TEMPORARY TABLE` statement, it does not `IF NOT EXISTS` for now.\n* For a `CREATE TABLE` statement (we are creating a metastore table), if there is an existing table having the same name ...\n  * when `IF NOT EXISTS` clause is used, we will do nothing.\n  * when `IF NOT EXISTS` clause is not used, the user will see an exception saying the table already exists.\n\nTODOs:\n- [x] CTAS support\n- [x] Programmatic APIs\n- [ ] Python API (another PR)\n- [x] More unit tests\n- [ ] Documents (another PR)\n\nmarmbrus liancheng rxin\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #4294 from yhuai/writeSupport and squashes the following commits:\n\n3db1539 [Yin Huai] save does not take overwrite.\n1c98881 [Yin Huai] Fix test.\n142372a [Yin Huai] Merge remote-tracking branch 'upstream/master' into writeSupport\n34e1bfb [Yin Huai] Address comments.\n1682ca6 [Yin Huai] Better support for CTAS statements.\ne789d64 [Yin Huai] For the Scala API, let users to use tuples to provide options.\n0128065 [Yin Huai] Short hand versions of save and load.\n66ebd74 [Yin Huai] Formatting.\n9203ec2 [Yin Huai] Merge remote-tracking branch 'upstream/master' into writeSupport\ne5d29f2 [Yin Huai] Programmatic APIs.\n1a719a5 [Yin Huai] CREATE TEMPORARY TABLE with IF NOT EXISTS is not allowed for now.\n909924f [Yin Huai] Add saveAsTable for the data source API to DataFrame.\n95a7c71 [Yin Huai] Fix bug when handling IF NOT EXISTS clause in a CREATE TEMPORARY TABLE statement.\nd37b19c [Yin Huai] Cheng's comments.\nfd6758c [Yin Huai] Use BeforeAndAfterAll.\n7880891 [Yin Huai] Support CREATE TABLE AS SELECT STATEMENT and the IF NOT EXISTS clause.\ncb85b05 [Yin Huai] Initial write support.\n2f91354 [Yin Huai] Make INSERT OVERWRITE/INTO statements consistent between HiveQL and SqlParser.","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/13531dd97c08563e53dacdaeaf1102bdd13ef825"},{"sha":"b8ebebeaaa259be4fcddf65b3280d23165b011a1","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-5414] Add SparkFirehoseListener class for consuming all SparkListener events\n\nThere isn't a good way to write a SparkListener that receives all SparkListener events and which will be future-compatible (e.g. it will receive events introduced in newer versions of Spark without having to override new methods to process those events).\n\nTo address this, this patch adds `SparkFirehoseListener`, a SparkListener implementation that receives all events and dispatches them to a single `onEvent` method (which can be overridden by users).\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #4210 from JoshRosen/firehose-listener and squashes the following commits:\n\n223f579 [Josh Rosen] Expand comment to explain rationale for this being a Java class.\necdfaed [Josh Rosen] Add SparkFirehoseListener class for consuming all SparkListener events.","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/b8ebebeaaa259be4fcddf65b3280d23165b011a1"},{"sha":"0cc7b88c99405db99bc4c3d66f5409e5da0e3c6e","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-5536] replace old ALS implementation by the new one\n\nThe only issue is that `analyzeBlock` is removed, which was marked as a developer API. I didn't change other tests in the ALSSuite under `spark.mllib` to ensure that the implementation is correct.\n\nCC: srowen coderxiang\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #4321 from mengxr/SPARK-5536 and squashes the following commits:\n\n5a3cee8 [Xiangrui Meng] update python tests that are too strict\ne840acf [Xiangrui Meng] ignore scala style check for ALS.train\ne9a721c [Xiangrui Meng] update mima excludes\n9ee6a36 [Xiangrui Meng] merge master\n9a8aeac [Xiangrui Meng] update tests\nd8c3271 [Xiangrui Meng] remove analyzeBlocks\nd68eee7 [Xiangrui Meng] add checkpoint to new ALS\n22a56f8 [Xiangrui Meng] wrap old ALS\nc387dff [Xiangrui Meng] support random seed\n3bdf24b [Xiangrui Meng] make storage level configurable in the new ALS","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/0cc7b88c99405db99bc4c3d66f5409e5da0e3c6e"},{"sha":"980764f3c0c065cc32454a036e8d0ead5a92037b","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-1405] [mllib] Latent Dirichlet Allocation (LDA) using EM\n\n**This PR introduces an API + simple implementation for Latent Dirichlet Allocation (LDA).**\n\nThe [design doc for this PR](https://docs.google.com/document/d/1kSsDqTeZMEB94Bs4GTd0mvdAmduvZSSkpoSfn-seAzo) has been updated since I initially posted it.  In particular, see the API and Planning for the Future sections.\n\n* Settle on a public API which may eventually include:\n  * more inference algorithms\n  * more options / functionality\n* Have an initial easy-to-understand implementation which others may improve.\n* This is NOT intended to support every topic model out there.  However, if there are suggestions for making this extensible or pluggable in the future, that could be nice, as long as it does not complicate the API or implementation too much.\n* This may not be very scalable currently.  It will be important to check and improve accuracy.  For correctness of the implementation, please check against the Asuncion et al. (2009) paper in the design doc.\n\n**Dependency: This makes MLlib depend on GraphX.**\n\nFiles and classes:\n* LDA.scala (441 lines):\n  * class LDA (main estimator class)\n  * LDA.Document  (text + document ID)\n* LDAModel.scala (266 lines)\n  * abstract class LDAModel\n  * class LocalLDAModel\n  * class DistributedLDAModel\n* LDAExample.scala (245 lines): script to run LDA + a simple (private) Tokenizer\n* LDASuite.scala (144 lines)\n\nData/model representation and algorithm:\n* Data/model: Uses GraphX, with term vertices + document vertices\n* Algorithm: EM, following [Asuncion, Welling, Smyth, and Teh.  \"On Smoothing and Inference for Topic Models.\"  UAI, 2009.](http://arxiv-web3.library.cornell.edu/abs/1205.2662v1)\n* For more details, please see the description in the “DEVELOPERS NOTE” in LDA.scala\n\nPlease refer to the JIRA for more discussion + the [design doc for this PR](https://docs.google.com/document/d/1kSsDqTeZMEB94Bs4GTd0mvdAmduvZSSkpoSfn-seAzo)\n\nHere, I list the main changes AFTER the design doc was posted.\n\nDesign decisions:\n* logLikelihood() computes the log likelihood of the data and the current point estimate of parameters.  This is different from the likelihood of the data given the hyperparameters, which would be harder to compute.  I’d describe the current approach as more frequentist, whereas the harder approach would be more Bayesian.\n* The current API takes Documents as token count vectors.  I believe there should be an extended API taking RDD[String] or RDD[Array[String]] in a future PR.  I have sketched this out in the design doc (as well as handier versions of getTopics returning Strings).\n* Hyperparameters should be set differently for different inference/learning algorithms.  See Asuncion et al. (2009) in the design doc for a good demonstration.  I encourage good behavior via defaults and warning messages.\n\nItems planned for future PRs:\n* perplexity\n* API taking Strings\n\n* Should LDA be called LatentDirichletAllocation (and LDAModel be LatentDirichletAllocationModel)?\n  * Pro: We may someday want LinearDiscriminantAnalysis.\n  * Con: Very long names\n\n* Should LDA reside in clustering?  Or do we want a sub-package?\n  * mllib.topicmodel\n  * mllib.clustering.topicmodel\n\n* Does the API seem reasonable and extensible?\n\n* Unit tests:\n  * Should there be a test which checks a clustering results?  E.g., train on a small, fake dataset with 2 very distinct topics/clusters, and ensure LDA finds those 2 topics/clusters.  Does that sound useful or too flaky?\n\nThis has not been tested much for scaling.  I have run it on a laptop for 200 iterations on a 5MB dataset with 1000 terms and 5 topics.  Running it for 500 iterations made it fail because of GC problems.  I'm running larger scale tests & will put results here, but future PRs may need to improve the scaling.\n\n* dlwh  for the initial implementation\n  * + jegonzal  for some code in the initial implementation\n* The many contributors towards topic model implementations in Spark which were referenced as a basis for this PR: akopich witgo yinxusen dlwh EntilZha jegonzal  IlyaKozlov\n  * Note: The plan is to include this full list in the authors if this PR gets merged.  Please notify me if you prefer otherwise.\n\nCC: mengxr\n\nAuthors:\n  Joseph K. Bradley <joseph@databricks.com>\n  Joseph Gonzalez <joseph.e.gonzalez@gmail.com>\n  David Hall <david.lw.hall@gmail.com>\n  Guoqiang Li <witgo@qq.com>\n  Xiangrui Meng <meng@databricks.com>\n  Pedro Rodriguez <pedro@snowgeek.org>\n  Avanesov Valeriy <acopich@gmail.com>\n  Xusen Yin <yinxusen@gmail.com>\n\nCloses #2388\nCloses #4047 from jkbradley/davidhall-lda and squashes the following commits:\n\n77e8814 [Joseph K. Bradley] small doc fix\n5c74345 [Joseph K. Bradley] cleaned up doc based on code review\n589728b [Joseph K. Bradley] Updates per code review.  Main change was in LDAExample for faster vocab computation.  Also updated PeriodicGraphCheckpointerSuite.scala to clean up checkpoint files at end\ne3980d2 [Joseph K. Bradley] cleaned up PeriodicGraphCheckpointerSuite.scala\n74487e5 [Joseph K. Bradley] Merge remote-tracking branch 'upstream/master' into davidhall-lda\n4ae2a7d [Joseph K. Bradley] removed duplicate graphx dependency in mllib/pom.xml\ne391474 [Joseph K. Bradley] Removed LDATiming.  Added PeriodicGraphCheckpointerSuite.scala.  Small LDA cleanups.\ne8d8acf [Joseph K. Bradley] Added catch for BreakIterator exception.  Improved preprocessing to reduce passes over data\n1a231b4 [Joseph K. Bradley] fixed scalastyle\n91aadfe [Joseph K. Bradley] Added Java-friendly run method to LDA. Added Java test suite for LDA. Changed LDAModel.describeTopics to return Java-friendly type\nb75472d [Joseph K. Bradley] merged improvements from LDATiming into LDAExample.  Will remove LDATiming after done testing\n993ca56 [Joseph K. Bradley] * Removed Document type in favor of (Long, Vector) * Changed doc ID restriction to be: id must be nonnegative and unique in the doc (instead of 0,1,2,...) * Add checks for valid ranges of eta, alpha * Rename “LearningState” to “EMOptimizer” * Renamed params: termSmoothing -> topicConcentration, topicSmoothing -> docConcentration   * Also added aliases alpha, beta\ncb5a319 [Joseph K. Bradley] Added checkpointing to LDA * new class PeriodicGraphCheckpointer * params checkpointDir, checkpointInterval to LDA\n43c1c40 [Joseph K. Bradley] small cleanup\n0b90393 [Joseph K. Bradley] renamed LDA LearningState.collectTopicTotals to globalTopicTotals\n77a2c85 [Joseph K. Bradley] Moved auto term,topic smoothing computation to get*Smoothing methods.  Changed word to term in some places.  Updated LDAExample to use default smoothing amounts.\nfb1e7b5 [Xiangrui Meng] minor\n08d59a3 [Xiangrui Meng] reset spacing\n9fe0b95 [Xiangrui Meng] optimize aggregateMessages\ncec0a9c [Xiangrui Meng] * -> *=\n6cb11b0 [Xiangrui Meng] optimize computePTopic\n9eb3d02 [Xiangrui Meng] + -> +=\n892530c [Xiangrui Meng] use axpy\n45cc7f2 [Xiangrui Meng] mapPart -> flatMap\nce53be9 [Joseph K. Bradley] fixed example name\n75749e7 [Joseph K. Bradley] scala style fix\n9f2a492 [Joseph K. Bradley] Unit tests and fixes for LDA, now ready for PR\n377ebd9 [Joseph K. Bradley] separated LDA models into own file.  more cleanups before PR\n2d40006 [Joseph K. Bradley] cleanups before PR\n2891e89 [Joseph K. Bradley] Prepped LDA main class for PR, but some cleanups remain\n0cb7187 [Joseph K. Bradley] Added 3 files from dlwh LDA implementation","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/980764f3c0c065cc32454a036e8d0ead5a92037b"},{"sha":"659329f9ee51ca8ae6232e07c45b5d9144d49667","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[minor] update streaming linear algorithms\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #4329 from mengxr/streaming-lr and squashes the following commits:\n\n78731e1 [Xiangrui Meng] update streaming linear algorithms","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/659329f9ee51ca8ae6232e07c45b5d9144d49667"},{"sha":"37df330135e6a3e62c580e5706eade5f1eaf5b13","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SQL][DataFrame] Remove DataFrameApi, ExpressionApi, and GroupedDataFrameApi\n\nThey were there mostly for code review and easier check of the API. I don't think they need to be there anymore.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #4328 from rxin/remove-df-api and squashes the following commits:\n\n723d600 [Reynold Xin] [SQL][DataFrame] Remove DataFrameApi and ColumnApi.","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/37df330135e6a3e62c580e5706eade5f1eaf5b13"},{"sha":"523a93523d0f9fc12de1ba2dc1acc360cdbc7027","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-5551][SQL] Create type alias for SchemaRDD for source backward compatibility\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #4327 from rxin/schemarddTypeAlias and squashes the following commits:\n\ne5a8ff3 [Reynold Xin] [SPARK-5551][SQL] Create type alias for SchemaRDD for source backward compatibility","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/523a93523d0f9fc12de1ba2dc1acc360cdbc7027"},{"sha":"bebf4c42bef3e75d31ffce9bfdb331c16f34ddb1","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-5549] Define TaskContext interface in Scala.\n\nSo the interface documentation shows up in ScalaDoc.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #4324 from rxin/TaskContext-scala and squashes the following commits:\n\n2480a17 [Reynold Xin] comment\n573756f [Reynold Xin] style fixes and javadoc fixes.\n87dd537 [Reynold Xin] [SPARK-5549] Define TaskContext interface in Scala.","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/bebf4c42bef3e75d31ffce9bfdb331c16f34ddb1"},{"sha":"f7948f3f5718b7c4a2d35634815670c4cbbe70fd","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Minor: Fix TaskContext deprecated annotations.\n\nMade a mistake in https://github.com/apache/spark/pull/4324\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #4333 from rxin/taskcontext-deprecate and squashes the following commits:\n\n61c44ee [Reynold Xin] Minor: Fix TaskContext deprecated annotations.","distinct":false,"url":"https://api.github.com/repos/jeanlyn/spark/commits/f7948f3f5718b7c4a2d35634815670c4cbbe70fd"}]}