{"push_id":650576705,"size":20,"distinct_size":10,"ref":"refs/heads/limit","head":"34c0514a10321e64078545b748dafda8c8b71d2c","before":"589e9aa0253e0e6d83b9e0164c869269390c619f","commits":[{"sha":"fa01bec484fc000e0a31645b722ffde48556c4df","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[Build] Enable MiMa checks for SQL\n\nNow that 1.3 has been released, we should enable MiMa checks for the `sql` subproject.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #5727 from JoshRosen/enable-more-mima-checks and squashes the following commits:\n\n3ad302b [Josh Rosen] Merge remote-tracking branch 'origin/master' into enable-more-mima-checks\n0c48e4d [Josh Rosen] Merge remote-tracking branch 'origin/master' into enable-more-mima-checks\ne276cee [Josh Rosen] Fix SQL MiMa checks via excludes and private[sql]\n44d0d01 [Josh Rosen] Add back 'launcher' exclude\n1aae027 [Josh Rosen] Enable MiMa checks for launcher and sql projects.","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/fa01bec484fc000e0a31645b722ffde48556c4df"},{"sha":"1c3e402e669d047410b00de9193adf3c329844a2","author":{"email":"2a07432b1b7252977a4c2dfc8e6d6d5ac763a480@netflix.com","name":"DB Tsai"},"message":"[SPARK-7279] Removed diffSum which is theoretical zero in LinearRegression and coding formating\n\nAuthor: DB Tsai <dbt@netflix.com>\n\nCloses #5809 from dbtsai/format and squashes the following commits:\n\n6904eed [DB Tsai] triger jenkins\n9146e19 [DB Tsai] initial commit","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/1c3e402e669d047410b00de9193adf3c329844a2"},{"sha":"149b3ee2dac992355adbe44e989570726c1f35d0","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":"[SPARK-7242][SQL][MLLIB] Frequent items for DataFrames\n\nFinding frequent items with possibly false positives, using the algorithm described in `http://www.cs.umd.edu/~samir/498/karp.pdf`.\npublic API under:\n```\ndf.stat.freqItems(cols: Array[String], support: Double = 0.001): DataFrame\n```\n\nThe output is a local DataFrame having the input column names with `-freqItems` appended to it. This is a single pass algorithm that may return false positives, but no false negatives.\n\ncc mengxr rxin\n\nLet's get the implementations in, I can add python API in a follow up PR.\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #5799 from brkyvz/freq-items and squashes the following commits:\n\na6ec82c [Burak Yavuz] addressed comments v?\n39b1bba [Burak Yavuz] removed toSeq\n0915e23 [Burak Yavuz] addressed comments v2.1\n3a5c177 [Burak Yavuz] addressed comments v2.0\n482e741 [Burak Yavuz] removed old import\n38e784d [Burak Yavuz] addressed comments v1.0\n8279d4d [Burak Yavuz] added default value for support\n3d82168 [Burak Yavuz] made base implementation","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/149b3ee2dac992355adbe44e989570726c1f35d0"},{"sha":"ee04413935f74b3178adbb6d8dee19b3320803e9","author":{"email":"53910baadf637a7954920bb7fe798d65a5bd4846@gmail.com","name":"rakeshchalasani"},"message":"[SPARK-7280][SQL] Add \"drop\" column/s on a data frame\n\nTakes a column name/s and returns a new DataFrame that drops a column/s.\n\nAuthor: rakeshchalasani <vnit.rakesh@gmail.com>\n\nCloses #5818 from rakeshchalasani/SPARK-7280 and squashes the following commits:\n\nce2ec09 [rakeshchalasani] Minor edit\n45c06f1 [rakeshchalasani] Change withColumnRename and format changes\nf68945a [rakeshchalasani] Minor fix\n0b9104d [rakeshchalasani] Drop one column at a time\n289afd2 [rakeshchalasani] [SPARK-7280][SQL] Add \"drop\" column/s on a data frame","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/ee04413935f74b3178adbb6d8dee19b3320803e9"},{"sha":"079733817f02c61ef814f5d9c0c8227498ff0058","author":{"email":"f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","name":"scwf"},"message":"[SPARK-7093] [SQL] Using newPredicate in NestedLoopJoin to enable code generation\n\nUsing newPredicate in NestedLoopJoin instead of InterpretedPredicate to make it can make use of code generation\n\nAuthor: scwf <wangfei1@huawei.com>\n\nCloses #5665 from scwf/NLP and squashes the following commits:\n\nd19dd31 [scwf] improvement\na887c02 [scwf] improve for NLP boundCondition","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/079733817f02c61ef814f5d9c0c8227498ff0058"},{"sha":"a0d8a61ab198b8c0ddbb3072bbe1d0e1dabc3e45","author":{"email":"f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","name":"wangfei"},"message":"[SPARK-7109] [SQL] Push down left side filter for left semi join\n\nNow in spark sql optimizer we only push down right side filter for left semi join, actually we can push down left side filter because left semi join is doing filter on left table essentially.\n\nAuthor: wangfei <wangfei1@huawei.com>\nAuthor: scwf <wangfei1@huawei.com>\n\nCloses #5677 from scwf/leftsemi and squashes the following commits:\n\n483d205 [wangfei] update with master to fix compile issue\n82df0e1 [wangfei] Merge branch 'master' of https://github.com/apache/spark into leftsemi\nd68a053 [wangfei] added apply\n8f48a3d [scwf] added test\nebadaa9 [wangfei] left filter push down for left semi join","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/a0d8a61ab198b8c0ddbb3072bbe1d0e1dabc3e45"},{"sha":"e991255e7203a0f7080efbd71f57574f46076711","author":{"email":"296ccda56873e68d94d02b8d7fd7c5480f0445e3@gmail.com","name":"Vyacheslav Baranov"},"message":"[SPARK-6913][SQL] Fixed \"java.sql.SQLException: No suitable driver found\"\n\nFixed `java.sql.SQLException: No suitable driver found` when loading DataFrame into Spark SQL if the driver is supplied with `--jars` argument.\n\nThe problem is in `java.sql.DriverManager` class that can't access drivers loaded by Spark ClassLoader.\n\nWrappers that forward requests are created for these drivers.\n\nAlso, it's not necessary any more to include JDBC drivers in `--driver-class-path` in local mode, specifying in `--jars` argument is sufficient.\n\nAuthor: Vyacheslav Baranov <slavik.baranov@gmail.com>\n\nCloses #5782 from SlavikBaranov/SPARK-6913 and squashes the following commits:\n\n510c43f [Vyacheslav Baranov] [SPARK-6913] Fixed review comments\nb2a727c [Vyacheslav Baranov] [SPARK-6913] Fixed thread race on driver registration\nc8294ae [Vyacheslav Baranov] [SPARK-6913] Fixed \"No suitable driver found\" when using using JDBC driver added with SparkContext.addJar","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/e991255e7203a0f7080efbd71f57574f46076711"},{"sha":"3ba5aaab8266822545ac82b9e733fd25cc215a77","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-5213] [SQL] Pluggable SQL Parser Support\n\nThis PR aims to make the SQL Parser Pluggable, and user can register it's own parser via Spark SQL CLI.\n\n```\n# add the jar into the classpath\n$hchengmydesktop:spark>bin/spark-sql --jars sql99.jar\n\n-- switch to \"hiveql\" dialect\n   spark-sql>SET spark.sql.dialect=hiveql;\n   spark-sql>SELECT * FROM src LIMIT 1;\n\n-- switch to \"sql\" dialect\n   spark-sql>SET spark.sql.dialect=sql;\n   spark-sql>SELECT * FROM src LIMIT 1;\n\n-- switch to a custom dialect\n   spark-sql>SET spark.sql.dialect=com.xxx.xxx.SQL99Dialect;\n   spark-sql>SELECT * FROM src LIMIT 1;\n\n-- register the non-exist SQL dialect\n   spark-sql> SET spark.sql.dialect=NotExistedClass;\n   spark-sql> SELECT * FROM src LIMIT 1;\n-- Exception will be thrown and switch to default sql dialect (\"sql\" for SQLContext and \"hiveql\" for HiveContext)\n```\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #4015 from chenghao-intel/sqlparser and squashes the following commits:\n\n493775c [Cheng Hao] update the code as feedback\n81a731f [Cheng Hao] remove the unecessary comment\naab0b0b [Cheng Hao] polish the code a little bit\n49b9d81 [Cheng Hao] shrink the comment for rebasing","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/3ba5aaab8266822545ac82b9e733fd25cc215a77"},{"sha":"473552fa5db9fa81f1a800f4ebacd23472e8c212","author":{"email":"f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","name":"scwf"},"message":"[SPARK-7123] [SQL] support table.star in sqlcontext\n\nRun following sql get error\n`SELECT r.*\nFROM testData l join testData2 r on (l.key = r.a)`\n\nAuthor: scwf <wangfei1@huawei.com>\n\nCloses #5690 from scwf/tablestar and squashes the following commits:\n\n3b2e2b6 [scwf] support table.star","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/473552fa5db9fa81f1a800f4ebacd23472e8c212"},{"sha":"beeafcfd6ee1e460c4d564cd1515d8781989b422","author":{"email":"cbb7353e6d953ef360baf960c122346276c6e320@databricks.com","name":"Patrick Wendell"},"message":"Revert \"[SPARK-5213] [SQL] Pluggable SQL Parser Support\"\n\nThis reverts commit 3ba5aaab8266822545ac82b9e733fd25cc215a77.","distinct":false,"url":"https://api.github.com/repos/scwf/spark/commits/beeafcfd6ee1e460c4d564cd1515d8781989b422"},{"sha":"69a739c7f5fd002432ece203957e1458deb2f4c3","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-7282] [STREAMING] Fix the race conditions in StreamingListenerSuite\n\nFixed the following flaky test\n```Scala\n[info] StreamingListenerSuite:\n[info] - batch info reporting (782 milliseconds)\n[info] - receiver info reporting *** FAILED *** (3 seconds, 911 milliseconds)\n[info]   The code passed to eventually never returned normally. Attempted 10 times over 3.4735783689999997 seconds. Last failure message: 0 did not equal 1. (StreamingListenerSuite.scala:104)\n[info]   org.scalatest.exceptions.TestFailedDueToTimeoutException:\n[info]   at org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420)\n[info]   at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438)\n[info]   at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)\n[info]   at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:307)\n[info]   at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2.apply$mcV$sp(StreamingListenerSuite.scala:104)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2.apply(StreamingListenerSuite.scala:94)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2.apply(StreamingListenerSuite.scala:94)\n[info]   at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)\n[info]   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)\n[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)\n[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)\n[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)\n[info]   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)\n[info]   at org.scalatest.Suite$class.withFixture(Suite.scala:1122)\n[info]   at org.scalatest.FunSuite.withFixture(FunSuite.scala:1555)\n[info]   at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)\n[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)\n[info]   at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite.org$scalatest$BeforeAndAfter$$super$runTest(StreamingListenerSuite.scala:34)\n[info]   at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:200)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite.runTest(StreamingListenerSuite.scala:34)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)\n[info]   at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)\n[info]   at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)\n[info]   at scala.collection.immutable.List.foreach(List.scala:318)\n[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)\n[info]   at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)\n[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)\n[info]   at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)\n[info]   at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)\n[info]   at org.scalatest.Suite$class.run(Suite.scala:1424)\n[info]   at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)\n[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:545)\n[info]   at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite.org$scalatest$BeforeAndAfter$$super$run(StreamingListenerSuite.scala:34)\n[info]   at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:241)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite.run(StreamingListenerSuite.scala:34)\n[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:462)\n[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:671)\n[info]   at sbt.ForkMain$Run$2.call(ForkMain.java:294)\n[info]   at sbt.ForkMain$Run$2.call(ForkMain.java:284)\n[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:262)\n[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n[info]   at java.lang.Thread.run(Thread.java:745)\n[info]   Cause: org.scalatest.exceptions.TestFailedException: 0 did not equal 1\n[info]   at org.scalatest.MatchersHelper$.newTestFailedException(MatchersHelper.scala:160)\n[info]   at org.scalatest.Matchers$ShouldMethodHelper$.shouldMatcher(Matchers.scala:6231)\n[info]   at org.scalatest.Matchers$AnyShouldWrapper.should(Matchers.scala:6277)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2$$anonfun$apply$mcV$sp$1.apply$mcV$sp(StreamingListenerSuite.scala:105)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2$$anonfun$apply$mcV$sp$1.apply(StreamingListenerSuite.scala:104)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2$$anonfun$apply$mcV$sp$1.apply(StreamingListenerSuite.scala:104)\n[info]   at org.scalatest.concurrent.Eventually$class.makeAValiantAttempt$1(Eventually.scala:394)\n[info]   at org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:408)\n[info]   at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438)\n[info]   at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)\n[info]   at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:307)\n[info]   at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2.apply$mcV$sp(StreamingListenerSuite.scala:104)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2.apply(StreamingListenerSuite.scala:94)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2.apply(StreamingListenerSuite.scala:94)\n[info]   at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)\n[info]   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)\n[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)\n[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)\n[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)\n[info]   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)\n[info]   at org.scalatest.Suite$class.withFixture(Suite.scala:1122)\n[info]   at org.scalatest.FunSuite.withFixture(FunSuite.scala:1555)\n[info]   at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)\n[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)\n[info]   at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite.org$scalatest$BeforeAndAfter$$super$runTest(StreamingListenerSuite.scala:34)\n[info]   at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:200)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite.runTest(StreamingListenerSuite.scala:34)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)\n[info]   at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)\n[info]   at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)\n[info]   at scala.collection.immutable.List.foreach(List.scala:318)\n[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)\n[info]   at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)\n[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)\n[info]   at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)\n[info]   at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)\n[info]   at org.scalatest.Suite$class.run(Suite.scala:1424)\n[info]   at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)\n[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:545)\n[info]   at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite.org$scalatest$BeforeAndAfter$$super$run(StreamingListenerSuite.scala:34)\n[info]   at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:241)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite.run(StreamingListenerSuite.scala:34)\n[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:462)\n[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:671)\n[info]   at sbt.ForkMain$Run$2.call(ForkMain.java:294)\n[info]   at sbt.ForkMain$Run$2.call(ForkMain.java:284)\n[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:262)\n[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n[info]   at java.lang.Thread.run(Thread.java:745)\n```\n\nThe original codes didn't have a memory barrier in the `eventually` closure, which might fail the test, because JVM doesn't guarantee the memory consistency between different threads without  a memory barrier.\n\nThis PR used `ConcurrentLinkedQueue` to set up the memory barrier.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #5812 from zsxwing/SPARK-7282 and squashes the following commits:\n\n59115ef [zsxwing] Use SynchronizedBuffer\n014dd2b [zsxwing] Fix the race conditions in StreamingListenerSuite","distinct":true,"url":"https://api.github.com/repos/scwf/spark/commits/69a739c7f5fd002432ece203957e1458deb2f4c3"},{"sha":"b5347a4664625ede6ab9d8ef6558457a34ae423f","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":" [SPARK-7248] implemented random number generators for DataFrames\n\nAdds the functions `rand` (Uniform Dist) and `randn` (Normal Dist.) as expressions to DataFrames.\n\ncc mengxr rxin\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #5819 from brkyvz/df-rng and squashes the following commits:\n\n50d69d4 [Burak Yavuz] add seed for test that failed\n4234c3a [Burak Yavuz] fix Rand expression\n13cad5c [Burak Yavuz] couple fixes\n7d53953 [Burak Yavuz] waiting for hive tests\nb453716 [Burak Yavuz] move radn with seed down\n03637f0 [Burak Yavuz] fix broken hive func\nc5909eb [Burak Yavuz] deleted old implementation of Rand\n6d43895 [Burak Yavuz] implemented random generators","distinct":true,"url":"https://api.github.com/repos/scwf/spark/commits/b5347a4664625ede6ab9d8ef6558457a34ae423f"},{"sha":"36a7a6807ee58c8ba309c0e09e547919246b7f82","author":{"email":"914a64bc573f63c3e7eba683b6cec5858e463b20@gmail.com","name":"Zhan Zhang"},"message":"[SPARK-6479] [BLOCK MANAGER] Create off-heap block storage API\n\nThis is the classes for creating off-heap block storage API. It also includes the migration for Tachyon. The diff seems to be big, but it mainly just rename tachyon to offheap. New implementation for hdfs will be submit for review in spark-6112.\n\nAuthor: Zhan Zhang <zhazhan@gmail.com>\n\nCloses #5430 from zhzhan/SPARK-6479 and squashes the following commits:\n\n60acd84 [Zhan Zhang] minor change to kickoff the test\n12f54c9 [Zhan Zhang] solve merge conflicts\na54132c [Zhan Zhang] solve review comments\nffb8e00 [Zhan Zhang] rebase to sparkcontext change\n6e121e0 [Zhan Zhang] resolve review comments and restructure blockmanasger code\na7aed6c [Zhan Zhang] add Tachyon migration code\n186de31 [Zhan Zhang] initial commit for off-heap block storage api","distinct":true,"url":"https://api.github.com/repos/scwf/spark/commits/36a7a6807ee58c8ba309c0e09e547919246b7f82"},{"sha":"a9fc50552ec96cd7817dfd19fc681b3368545ee3","author":{"email":"cbb7353e6d953ef360baf960c122346276c6e320@databricks.com","name":"Patrick Wendell"},"message":"HOTFIX: Disable buggy dependency checker","distinct":true,"url":"https://api.github.com/repos/scwf/spark/commits/a9fc50552ec96cd7817dfd19fc681b3368545ee3"},{"sha":"0a2b15ce43cf6096e1a7ae060b7c8a4010ce3b92","author":{"email":"5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","name":"Sandy Ryza"},"message":"[SPARK-4550] In sort-based shuffle, store map outputs in serialized form\n\nRefer to the JIRA for the design doc and some perf results.\n\nI wanted to call out some of the more possibly controversial changes up front:\n* Map outputs are only stored in serialized form when Kryo is in use.  I'm still unsure whether Java-serialized objects can be relocated.  At the very least, Java serialization writes out a stream header which causes problems with the current approach, so I decided to leave investigating this to future work.\n* The shuffle now explicitly operates on key-value pairs instead of any object.  Data is written to shuffle files in alternating keys and values instead of key-value tuples.  `BlockObjectWriter.write` now accepts a key argument and a value argument instead of any object.\n* The map output buffer can hold a max of Integer.MAX_VALUE bytes.  Though this wouldn't be terribly difficult to change.\n* When spilling occurs, the objects that still in memory at merge time end up serialized and deserialized an extra time.\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #4450 from sryza/sandy-spark-4550 and squashes the following commits:\n\n8c70dd9 [Sandy Ryza] Fix serialization\n9c16fe6 [Sandy Ryza] Fix a couple tests and move getAutoReset to KryoSerializerInstance\n6c54e06 [Sandy Ryza] Fix scalastyle\nd8462d8 [Sandy Ryza] SPARK-4550","distinct":true,"url":"https://api.github.com/repos/scwf/spark/commits/0a2b15ce43cf6096e1a7ae060b7c8a4010ce3b92"},{"sha":"7cf1eb79b1fa290aa1d867a8a1eaaea86d6b2239","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":"[SPARK-7287] enabled fixed test\n\nandrewor14 pwendell I reenabled the test. Let's see if it's fixed. I did also notice that `--jars` started to fail after this was ignored though in the JIRA. like [here](https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.0,label=centos/2238/consoleFull), you see that --jars fails for the same exact reason.\n\nHas any change been made to Spark Submit recently? Did the test setup on Jenkins change? If we look into flaky tests last month, you wouldn't find this test among them.\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #5826 from brkyvz/restart-test and squashes the following commits:\n\nf509f68 [Burak Yavuz] enabled fixed test","distinct":true,"url":"https://api.github.com/repos/scwf/spark/commits/7cf1eb79b1fa290aa1d867a8a1eaaea86d6b2239"},{"sha":"14b32886fa01aef6cc0dfbc263eb6d4c9d2876fa","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-7291] [CORE] Fix a flaky test in AkkaRpcEnvSuite\n\nRead the port from RpcEnv to check the result so that it will success even if port conflicts\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #5822 from zsxwing/SPARK-7291 and squashes the following commits:\n\ne521b84 [zsxwing] Fix a flaky test in AkkaRpcEnvSuite","distinct":true,"url":"https://api.github.com/repos/scwf/spark/commits/14b32886fa01aef6cc0dfbc263eb6d4c9d2876fa"},{"sha":"c24aeb6a310b49dba8db1f4642531780a2e27253","author":{"email":"8757c4a255384de9fc2175f17a84f948e7081d30@gmail.com","name":"MechCoder"},"message":"[SPARK-6257] [PYSPARK] [MLLIB] MLlib API missing items in Recommendation\n\nAdds\n\nrank, recommendUsers and RecommendProducts to MatrixFactorizationModel in PySpark.\n\nAuthor: MechCoder <manojkumarsivaraj334@gmail.com>\n\nCloses #5807 from MechCoder/spark-6257 and squashes the following commits:\n\n09629c6 [MechCoder] doc\n953b326 [MechCoder] [SPARK-6257] MLlib API missing items in Recommendation","distinct":true,"url":"https://api.github.com/repos/scwf/spark/commits/c24aeb6a310b49dba8db1f4642531780a2e27253"},{"sha":"7fe0f3f2b46c61a5cc4af9166781624409fda8a4","author":{"email":"7d45abe39fd242ee2f6575da28f13d66568418b3@oss.nttdata.co.jp","name":"Kousuke Saruta"},"message":"[SPARK-3468] [WEBUI] Timeline-View feature\n\nI sometimes trouble-shoot and analyse the cause of long time spending job.\n\nAt the time, I find the stages which spends long time or fails, then I find the tasks which spends long time or fails, next I analyse the proportion of each phase in a task.\n\nAnother case, I find executors which spends long time for running a task and analyse the details of a task.\n\nIn such situation, I think it's helpful to visualize timeline  for each application, job, task and the details of proportion of activity for each task.\n\nI added 3 timeline features to existing Web UI.\n\n[Application Timeline View]\nThis view shows following things.\n\n* When each executor was added/removed and the reason why it's removed.\n* When each job  was started/finished.\n* Status of each job.\n\n![screenshot from 2015-04-01 16 49 25](https://cloud.githubusercontent.com/assets/4736016/6936886/e35fd582-d891-11e4-980d-8de13f50e442.png)\n\n[Stage Timeline View]\nSimilar to Application Timeline View, this view shows following things.\n\n* When each executor was added/removed and the reason why it's removed.\n* When each job was started/finished.\n* Status of each stage.\n\n![screenshot from 2015-04-01 16 50 59](https://cloud.githubusercontent.com/assets/4736016/6936900/0dca6526-d892-11e4-84a8-efd9037af444.png)\n\n[Task Assignment Timeline View]\nThis view shows following things.\n\n* When each task started/finished\n* How long each task spent and the proportion.\n* Status of each task.\n* Where each task ran on.\n\n![screenshot from 2015-04-01 16 51 54](https://cloud.githubusercontent.com/assets/4736016/6936910/20fd5acc-d892-11e4-9018-80e463881fc2.png)\n\nAll the view above is zoomable by mouse wheel action and scrollable by drag action.\n\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #2342 from sarutak/timeline-viewer-feature and squashes the following commits:\n\n11fe67d [Kousuke Saruta] Fixed conflict\n79ac03d [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature\na91abd3 [Kousuke Saruta] Merge branch 'master' of https://github.com/apache/spark into timeline-viewer-feature\nef34a5b [Kousuke Saruta] Implement tooltip using bootstrap\nb09d0c5 [Kousuke Saruta] Move `stroke` and `fill` attribute of rect elements to css\nd3c63c8 [Kousuke Saruta] Fixed a little bit bugs\na36291b [Kousuke Saruta] Merge branch 'master' of https://github.com/apache/spark into timeline-viewer-feature\n28714b6 [Kousuke Saruta] Fixed highlight issue\n0dc4278 [Kousuke Saruta] Addressed most of Patrics's feedbacks\n8110acf [Kousuke Saruta] Added scroll limit to Job timeline\n974a64a [Kousuke Saruta] Removed unused function\nee7a7f0 [Kousuke Saruta] Refactored\n6a91872 [Kousuke Saruta] Temporary commit\n6693f34 [Kousuke Saruta] Added link to job/stage box in the timeline in order to move to corresponding row when we click\n8f88222 [Kousuke Saruta] Added job/stage description\naeed4b1 [Kousuke Saruta] Removed stage timeline\nfc1696c [Kousuke Saruta] Merge branch 'timeline-viewer-feature' of github.com:sarutak/spark into timeline-viewer-feature\n999ccd4 [Kousuke Saruta] Improved scalability\n0fc6a31 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature\n19815ae [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature\n68b7540 [Kousuke Saruta] Merge branch 'timeline-viewer-feature' of github.com:sarutak/spark into timeline-viewer-feature\n52b5f0b [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature\ndec85db [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature\nfcdab7d [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature\ndab7cc1 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature\n09cce97 [Kousuke Saruta] Cleanuped\n16f82cf [Kousuke Saruta] Cleanuped\n9fb522e [Kousuke Saruta] Cleanuped\nd05f2c2 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature\ne85e9aa [Kousuke Saruta] Cleanup: Added TimelineViewUtils.scala\na76e569 [Kousuke Saruta] Removed unused setting in timeline-view.css\n5ce1b21 [Kousuke Saruta] Added vis.min.js, vis.min.css and vis.map to .rat-exclude\n082f709 [Kousuke Saruta] Added Timeline-View feature for Applications, Jobs and Stages","distinct":true,"url":"https://api.github.com/repos/scwf/spark/commits/7fe0f3f2b46c61a5cc4af9166781624409fda8a4"},{"sha":"34c0514a10321e64078545b748dafda8c8b71d2c","author":{"email":"f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","name":"scwf"},"message":"Merge branch 'master' of https://github.com/apache/spark into limit","distinct":true,"url":"https://api.github.com/repos/scwf/spark/commits/34c0514a10321e64078545b748dafda8c8b71d2c"}]}