{"push_id":800349640,"size":12,"distinct_size":12,"ref":"refs/heads/master","head":"0494c80ef54f6f3a8c6f2d92abfe1a77a91df8b0","before":"01440395176bdbb2662480f03b27851cb860f385","commits":[{"sha":"20a61dbd9b57957fcc5b58ef8935533914172b07","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-10626] [MLLIB] create java friendly method for random rdd\n\nSPARK-3136 added a large number of functions for creating Java RandomRDDs, but for people that want to use custom RandomDataGenerators we should make a Java friendly method.\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #8782 from holdenk/SPARK-10626-create-java-friendly-method-for-randomRDD.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/20a61dbd9b57957fcc5b58ef8935533914172b07"},{"sha":"ebbf85f07bb8de0d566f1ae4b41f26421180bebe","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-7989] [SPARK-10651] [CORE] [TESTS] Increase timeout to fix flaky tests\n\nI noticed only one block manager registered with master in an unsuccessful build (https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.2,label=spark-test/3534/)\n```\n15/09/16 13:02:30.981 pool-1-thread-1-ScalaTest-running-BroadcastSuite INFO SparkContext: Running Spark version 1.6.0-SNAPSHOT\n...\n15/09/16 13:02:38.133 sparkDriver-akka.actor.default-dispatcher-19 INFO BlockManagerMasterEndpoint: Registering block manager localhost:48196 with 530.3 MB RAM, BlockManagerId(0, localhost, 48196)\n```\nIn addition, the first block manager needed 7+ seconds to start. But the test expected 2 block managers so it failed.\n\nHowever, there was no exception in this log file. So I checked a successful build (https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/3536/AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.2,label=spark-test/) and it needed 4-5 seconds to set up the local cluster:\n```\n15/09/16 18:11:27.738 sparkWorker1-akka.actor.default-dispatcher-5 INFO Worker: Running Spark version 1.6.0-SNAPSHOT\n...\n15/09/16 18:11:30.838 sparkDriver-akka.actor.default-dispatcher-20 INFO BlockManagerMasterEndpoint: Registering block manager localhost:54202 with 530.3 MB RAM, BlockManagerId(1, localhost, 54202)\n15/09/16 18:11:32.112 sparkDriver-akka.actor.default-dispatcher-20 INFO BlockManagerMasterEndpoint: Registering block manager localhost:32955 with 530.3 MB RAM, BlockManagerId(0, localhost, 32955)\n```\nIn this build, the first block manager needed only 3+ seconds to start.\n\nComparing these two builds, I guess it's possible that the local cluster in `BroadcastSuite` cannot be ready in 10 seconds if the Jenkins worker is busy. So I just increased the timeout to 60 seconds to see if this can fix the issue.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #8813 from zsxwing/fix-BroadcastSuite.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/ebbf85f07bb8de0d566f1ae4b41f26421180bebe"},{"sha":"ca9fe540fe04e2e230d1e76526b5502bab152914","author":{"email":"5cafd55dd0ca8119e8e1d62a6779a4b9bbdd477d@deepsense.io","name":"Jacek Laskowski"},"message":"[SPARK-10662] [DOCS] Code snippets are not properly formatted in tables\n\n* Backticks are processed properly in Spark Properties table\n* Removed unnecessary spaces\n* See http://people.apache.org/~pwendell/spark-nightly/spark-master-docs/latest/running-on-yarn.html\n\nAuthor: Jacek Laskowski <jacek.laskowski@deepsense.io>\n\nCloses #8795 from jaceklaskowski/docs-yarn-formatting.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/ca9fe540fe04e2e230d1e76526b5502bab152914"},{"sha":"331f0b10f78a37d96d3e573d211d74a0935265db","author":{"email":"e28226ca06290852b9d5dc31e037df691e0b2911@umich.edu","name":"Meihua Wu"},"message":"[SPARK-9642] [ML] LinearRegression should supported weighted data\n\nIn many modeling application, data points are not necessarily sampled with equal probabilities. Linear regression should support weighting which account the over or under sampling.\n\nwork in progress.\n\nAuthor: Meihua Wu <meihuawu@umich.edu>\n\nCloses #8631 from rotationsymmetry/SPARK-9642.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/331f0b10f78a37d96d3e573d211d74a0935265db"},{"sha":"b78c65b03ae87a3ba348c9d29ff4c296349eb49c","author":{"email":"f4cfeb6da63cd4b62a6340d67bbf18015ad432f2@xiaomi.com","name":"hushan[胡珊]"},"message":"[SPARK-5259] [CORE] don't submit stage until its dependencies map outputs are registered\n\nTrack pending tasks by partition ID instead of Task objects.\n\nBefore this change, failure & retry could result in a case where a stage got submitted before the map output from its dependencies get registered.  This was due to an error in the condition for registering map outputs.\n\nAuthor: hushan[胡珊] <hushan@xiaomi.com>\nAuthor: Imran Rashid <irashid@cloudera.com>\n\nCloses #7699 from squito/SPARK-5259.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/b78c65b03ae87a3ba348c9d29ff4c296349eb49c"},{"sha":"ba882db6f43dd2bc05675133158e4664ed07030a","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-9769] [ML] [PY] add python api for countvectorizermodel\n\nFrom JIRA: Add Python API, user guide and example for ml.feature.CountVectorizerModel\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #8561 from holdenk/SPARK-9769-add-python-api-for-countvectorizermodel.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/ba882db6f43dd2bc05675133158e4664ed07030a"},{"sha":"aeef44a3e32b53f7adecc8e9cfd684fb4598e87d","author":{"email":"9835e449f9a666ddd8ba8aeb460af39a5130691b@databricks.com","name":"Feynman Liang"},"message":"[SPARK-3147] [MLLIB] [STREAMING] Streaming 2-sample statistical significance testing\n\nImplementation of significance testing using Streaming API.\n\nAuthor: Feynman Liang <fliang@databricks.com>\nAuthor: Feynman Liang <feynman.liang@gmail.com>\n\nCloses #4716 from feynmanliang/ab_testing.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/aeef44a3e32b53f7adecc8e9cfd684fb4598e87d"},{"sha":"97a99dde6e8d69a4c4c135dc1d9b1520b2548b5b","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-10676] [DOCS] Add documentation for SASL encryption options.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #8803 from vanzin/SPARK-10676.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/97a99dde6e8d69a4c4c135dc1d9b1520b2548b5b"},{"sha":"362539f8d97f6bb67f0d0983f7dea36b77cc9d18","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-10630] [SQL] Add a createDataFrame API that takes in a java list\n\nIt would be nice to support creating a DataFrame directly from a Java List of Row.\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #8779 from holdenk/SPARK-10630-create-DataFrame-from-Java-List.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/362539f8d97f6bb67f0d0983f7dea36b77cc9d18"},{"sha":"7c4f852bfc39537840f56cd8121457a0dc1ad7c1","author":{"email":"1d6e1cf70ec6f9ab28d3ea4b27a49a77654d370e@noelsmith.com","name":"noelsmith"},"message":"[DOC] [PYSPARK] [MLLIB] Added newlines to docstrings to fix parameter formatting\n\nAdded newlines before `:param ...:` and `:return:` markup. Without these, parameter lists aren't formatted correctly in the API docs. I.e:\n\n![screen shot 2015-09-21 at 21 49 26](https://cloud.githubusercontent.com/assets/11915197/10004686/de3c41d4-60aa-11e5-9c50-a46dcb51243f.png)\n\n.. looks like this once newline is added:\n\n![screen shot 2015-09-21 at 21 50 14](https://cloud.githubusercontent.com/assets/11915197/10004706/f86bfb08-60aa-11e5-8524-ae4436713502.png)\n\nAuthor: noelsmith <mail@noelsmith.com>\n\nCloses #8851 from noel-smith/docstring-missing-newline-fix.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/7c4f852bfc39537840f56cd8121457a0dc1ad7c1"},{"sha":"72869883f12b6e0a4e5aad79c0ac2cfdb4d83f09","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-10649] [STREAMING] Prevent inheriting job group and irrelevant job description in streaming jobs\n\nThe job group, and job descriptions information is passed through thread local properties, and get inherited by child threads. In case of spark streaming, the streaming jobs inherit these properties from the thread that called streamingContext.start(). This may not make sense.\n\n1. Job group: This is mainly used for cancelling a group of jobs together. It does not make sense to cancel streaming jobs like this, as the effect will be unpredictable. And its not a valid usecase any way, to cancel a streaming context, call streamingContext.stop()\n\n2. Job description: This is used to pass on nice text descriptions for jobs to show up in the UI. The job description of the thread that calls streamingContext.start() is not useful for all the streaming jobs, as it does not make sense for all of the streaming jobs to have the same description, and the description may or may not be related to streaming.\n\nThe solution in this PR is meant for the Spark master branch, where local properties are inherited by cloning the properties. The job group and job description in the thread that starts the streaming scheduler are explicitly removed, so that all the subsequent child threads does not inherit them. Also, the starting is done in a new child thread, so that setting the job group and description for streaming, does not change those properties in the thread that called streamingContext.start().\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #8781 from tdas/SPARK-10649.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/72869883f12b6e0a4e5aad79c0ac2cfdb4d83f09"},{"sha":"0494c80ef54f6f3a8c6f2d92abfe1a77a91df8b0","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-10495] [SQL] Read date values in JSON data stored by Spark 1.5.0.\n\nhttps://issues.apache.org/jira/browse/SPARK-10681\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #8806 from yhuai/SPARK-10495.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/0494c80ef54f6f3a8c6f2d92abfe1a77a91df8b0"}]}