{"push_id":652064338,"size":93,"distinct_size":36,"ref":"refs/heads/Sent2vec","head":"01637e653c7175f731d4f54b4840f3d88d6a2515","before":"7714f0f02291742624241aafa34316df79260750","commits":[{"sha":"7dacc08ab36188991a001df23880167433844767","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":"[SPARK-7224] added mock repository generator for --packages tests\n\nThis patch contains an `IvyTestUtils` file, which dynamically generates jars and pom files to test the `--packages` feature without having to rely on the internet, and Maven Central.\n\ncc pwendell I know that there existed Util functions to create Jars and stuff already, but they didn't really serve my purposes as they appended random prefixes that was breaking things.\n\nI also added the local repository tests. Notice that they work without passing the `repo` to `resolveMavenCoordinates`.\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #5790 from brkyvz/maven-utils and squashes the following commits:\n\n3ec79b7 [Burak Yavuz] addressed comments v0.2\na39151b [Burak Yavuz] address comments v0.1\n172dfef [Burak Yavuz] use Ivy format\n7476d06 [Burak Yavuz] added mock repository generator","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/7dacc08ab36188991a001df23880167433844767"},{"sha":"6c65da6bb7d1213e6a4a9f7fd1597d029d87d07c","author":{"email":"e1f8d9790f67825b23faf6a373ab7c83bc8900f2@apache.org","name":"Hari Shreedharan"},"message":"[SPARK-5342] [YARN] Allow long running Spark apps to run on secure YARN/HDFS\n\nCurrent Spark apps running on Secure YARN/HDFS would not be able to write data\nto HDFS after 7 days, since delegation tokens cannot be renewed beyond that. This\nmeans Spark Streaming apps will not be able to run on Secure YARN.\n\nThis commit adds basic functionality to fix this issue. In this patch:\n- new parameters are added - principal and keytab, which can be used to login to a KDC\n- the client logs in, and then get tokens to start the AM\n- the keytab is copied to the staging directory\n- the AM waits for 60% of the time till expiry of the tokens and then logs in using the keytab\n- each time after 60% of the time, new tokens are created and sent to the executors\n\nCurrently, to avoid complicating the architecture, we set the keytab and principal in the\nSparkHadoopUtil singleton, and schedule a login. Once the login is completed, a callback is scheduled.\n\nThis is being posted for feedback, so I can gather feedback on the general implementation.\n\nThere are currently a bunch of things to do:\n- [x] logging\n- [x] testing - I plan to manually test this soon. If you have ideas of how to add unit tests, comment.\n- [x] add code to ensure that if these params are set in non-YARN cluster mode, we complain\n- [x] documentation\n- [x] Have the executors request for credentials from the AM, so that retries are possible.\n\nAuthor: Hari Shreedharan <hshreedharan@apache.org>\n\nCloses #4688 from harishreedharan/kerberos-longrunning and squashes the following commits:\n\n36eb8a9 [Hari Shreedharan] Change the renewal interval config param. Fix a bunch of comments.\n611923a [Hari Shreedharan] Make sure the namenodes are listed correctly for creating tokens.\n09fe224 [Hari Shreedharan] Use token.renew to get token's renewal interval rather than using hdfs-site.xml\n6963bbc [Hari Shreedharan] Schedule renewal in AM before starting user class. Else, a restarted AM cannot access HDFS if the user class tries to.\n072659e [Hari Shreedharan] Fix build failure caused by thread factory getting moved to ThreadUtils.\nf041dd3 [Hari Shreedharan] Merge branch 'master' into kerberos-longrunning\n42eead4 [Hari Shreedharan] Remove RPC part. Refactor and move methods around, use renewal interval rather than max lifetime to create new tokens.\nebb36f5 [Hari Shreedharan] Merge branch 'master' into kerberos-longrunning\nbc083e3 [Hari Shreedharan] Overload RegisteredExecutor to send tokens. Minor doc updates.\n7b19643 [Hari Shreedharan] Merge branch 'master' into kerberos-longrunning\n8a4f268 [Hari Shreedharan] Added docs in the security guide. Changed some code to ensure that the renewer objects are created only if required.\ne800c8b [Hari Shreedharan] Restore original RegisteredExecutor message, and send new tokens via NewTokens message.\n0e9507e [Hari Shreedharan] Merge branch 'master' into kerberos-longrunning\n7f1bc58 [Hari Shreedharan] Minor fixes, cleanup.\nbcd11f9 [Hari Shreedharan] Refactor AM and Executor token update code into separate classes, also send tokens via akka on executor startup.\nf74303c [Hari Shreedharan] Move the new logic into specialized classes. Add cleanup for old credentials files.\n2f9975c [Hari Shreedharan] Ensure new tokens are written out immediately on AM restart. Also, pikc up the latest suffix from HDFS if the AM is restarted.\n61b2b27 [Hari Shreedharan] Account for AM restarts by making sure lastSuffix is read from the files on HDFS.\n62c45ce [Hari Shreedharan] Relogin from keytab periodically.\nfa233bd [Hari Shreedharan] Adding logging, fixing minor formatting and ordering issues.\n42813b4 [Hari Shreedharan] Remove utils.sh, which was re-added due to merge with master.\n0de27ee [Hari Shreedharan] Merge branch 'master' into kerberos-longrunning\n55522e3 [Hari Shreedharan] Fix failure caused by Preconditions ambiguity.\n9ef5f1b [Hari Shreedharan] Added explanation of how the credentials refresh works, some other minor fixes.\nf4fd711 [Hari Shreedharan] Fix SparkConf usage.\n2debcea [Hari Shreedharan] Change the file structure for credentials files. I will push a followup patch which adds a cleanup mechanism for old credentials files. The credentials files are small and few enough for it to cause issues on HDFS.\naf6d5f0 [Hari Shreedharan] Cleaning up files where changes weren't required.\nf0f54cb [Hari Shreedharan] Be more defensive when updating the credentials file.\nf6954da [Hari Shreedharan] Got rid of Akka communication to renew, instead the executors check a known file's modification time to read the credentials.\n5c11c3e [Hari Shreedharan] Move tests to YarnSparkHadoopUtil to fix compile issues.\nb4cb917 [Hari Shreedharan] Send keytab to AM via DistributedCache rather than directly via HDFS\n0985b4e [Hari Shreedharan] Write tokens to HDFS and read them back when required, rather than sending them over the wire.\nd79b2b9 [Hari Shreedharan] Make sure correct credentials are passed to FileSystem#addDelegationTokens()\n8c6928a [Hari Shreedharan] Fix issue caused by direct creation of Actor object.\nfb27f46 [Hari Shreedharan] Make sure principal and keytab are set before CoarseGrainedSchedulerBackend is started. Also schedule re-logins in CoarseGrainedSchedulerBackend#start()\n41efde0 [Hari Shreedharan] Merge branch 'master' into kerberos-longrunning\nd282d7a [Hari Shreedharan] Fix ClientSuite to set YARN mode, so that the correct class is used in tests.\nbcfc374 [Hari Shreedharan] Fix Hadoop-1 build by adding no-op methods in SparkHadoopUtil, with impl in YarnSparkHadoopUtil.\nf8fe694 [Hari Shreedharan] Handle None if keytab-login is not scheduled.\n2b0d745 [Hari Shreedharan] [SPARK-5342][YARN] Allow long running Spark apps to run on secure YARN/HDFS.\nccba5bc [Hari Shreedharan] WIP: More changes wrt kerberos\n77914dd [Hari Shreedharan] WIP: Add kerberos principal and keytab to YARN client.","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/6c65da6bb7d1213e6a4a9f7fd1597d029d87d07c"},{"sha":"adbdb19a7d2cc939795f0cecbdc07c605dc946c1","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-7207] [ML] [BUILD] Added ml.recommendation, ml.regression to SparkBuild\n\nAdded ml.recommendation, ml.regression to SparkBuild\n\nCC: mengxr\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #5758 from jkbradley/SPARK-7207 and squashes the following commits:\n\na28158a [Joseph K. Bradley] Added ml.recommendation, ml.regression to SparkBuild","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/adbdb19a7d2cc939795f0cecbdc07c605dc946c1"},{"sha":"e0628f2fae7f99d096f9dd625876a60d11020d9b","author":{"email":"cbb7353e6d953ef360baf960c122346276c6e320@databricks.com","name":"Patrick Wendell"},"message":"Revert \"[SPARK-5342] [YARN] Allow long running Spark apps to run on secure YARN/HDFS\"\n\nThis reverts commit 6c65da6bb7d1213e6a4a9f7fd1597d029d87d07c.","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/e0628f2fae7f99d096f9dd625876a60d11020d9b"},{"sha":"6702324b60f99dab55912c08ccd3d03805f6b7b2","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-7196][SQL] Support precision and scale of decimal type for JDBC\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-7196\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #5777 from viirya/jdbc_precision and squashes the following commits:\n\nf40f5e6 [Liang-Chi Hsieh] Support precision and scale for NUMERIC type.\n49acbf9 [Liang-Chi Hsieh] Add unit test.\na509e19 [Liang-Chi Hsieh] Support precision and scale of decimal type for JDBC.","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/6702324b60f99dab55912c08ccd3d03805f6b7b2"},{"sha":"07a86205f9efc43ea1ec5edb97c21c32abe7fb8a","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-7288] Suppress compiler warnings due to use of sun.misc.Unsafe; add facade in front of Unsafe; remove use of Unsafe.setMemory\n\nThis patch suppresses compiler warnings due to our use of `sun.misc.Unsafe` (introduced in #5725).  These warnings can only be suppressed via the `-XDignore.symbol.file` javac flag; the `SuppressWarnings` annotation won't work for these.\n\nIn order to restrict uses of this compiler flag to the `unsafe` module, I placed a facade in front of `Unsafe` so that other modules won't call it directly. This facade also will also help us to avoid accidental usage of deprecated Unsafe methods or methods that aren't supported in Java 6.\n\nI also removed an unnecessary use of `Unsafe.setMemory`, which isn't present in certain versions of Java 6, and excluded the new `unsafe` module from Javadoc.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #5814 from JoshRosen/unsafe-compiler-warnings-fixes and squashes the following commits:\n\n9e8c483 [Josh Rosen] Exclude new unsafe module from Javadoc\nba75ecf [Josh Rosen] Only apply -XDignore.symbol.file flag in unsafe project.\n7403345 [Josh Rosen] Put facade in front of Unsafe.\n50230c0 [Josh Rosen] Remove usage of Unsafe.setMemory\n96d41c9 [Josh Rosen] Use -XDignore.symbol.file to suppress warnings about sun.misc.Unsafe usage","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/07a86205f9efc43ea1ec5edb97c21c32abe7fb8a"},{"sha":"77cc25fb7473d8a06b727d2ba5ee62db1c651cf8","author":{"email":"7acf0e49bc9acc50da7e38d41884e81d5f2df305@qq.com","name":"Zhongshuai Pei"},"message":"[SPARK-7267][SQL]Push down Project when it's child is Limit\n\nSQL\n```\nselect key from (select key,value from t1 limit 100) t2 limit 10\n```\nOptimized Logical Plan before modifying\n```\n== Optimized Logical Plan ==\nLimit 10\n  Project key#228\n    Limit 100\n      MetastoreRelation default, t1, None\n```\nOptimized Logical Plan after modifying\n```\n== Optimized Logical Plan ==\nLimit 10\n  Limit 100\n    Project key#228\n      MetastoreRelation default, t1, None\n```\nAfter this, we can combine limits\n\nAuthor: Zhongshuai Pei <799203320@qq.com>\nAuthor: DoingDone9 <799203320@qq.com>\n\nCloses #5797 from DoingDone9/ProjectLimit and squashes the following commits:\n\n70d0fca [Zhongshuai Pei] Update FilterPushdownSuite.scala\ndc83ae9 [Zhongshuai Pei] Update FilterPushdownSuite.scala\n485c61c [Zhongshuai Pei] Update Optimizer.scala\nf03fe7f [Zhongshuai Pei] Merge pull request #12 from apache/master\nf12fa50 [Zhongshuai Pei] Merge pull request #10 from apache/master\nf61210c [Zhongshuai Pei] Merge pull request #9 from apache/master\n34b1a9a [Zhongshuai Pei] Merge pull request #8 from apache/master\n802261c [DoingDone9] Merge pull request #7 from apache/master\nd00303b [DoingDone9] Merge pull request #6 from apache/master\n98b134f [DoingDone9] Merge pull request #5 from apache/master\n161cae3 [DoingDone9] Merge pull request #4 from apache/master\nc87e8b6 [DoingDone9] Merge pull request #3 from apache/master\ncb1852d [DoingDone9] Merge pull request #2 from apache/master\nc3f046f [DoingDone9] Merge pull request #1 from apache/master","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/77cc25fb7473d8a06b727d2ba5ee62db1c651cf8"},{"sha":"fa01bec484fc000e0a31645b722ffde48556c4df","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[Build] Enable MiMa checks for SQL\n\nNow that 1.3 has been released, we should enable MiMa checks for the `sql` subproject.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #5727 from JoshRosen/enable-more-mima-checks and squashes the following commits:\n\n3ad302b [Josh Rosen] Merge remote-tracking branch 'origin/master' into enable-more-mima-checks\n0c48e4d [Josh Rosen] Merge remote-tracking branch 'origin/master' into enable-more-mima-checks\ne276cee [Josh Rosen] Fix SQL MiMa checks via excludes and private[sql]\n44d0d01 [Josh Rosen] Add back 'launcher' exclude\n1aae027 [Josh Rosen] Enable MiMa checks for launcher and sql projects.","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/fa01bec484fc000e0a31645b722ffde48556c4df"},{"sha":"1c3e402e669d047410b00de9193adf3c329844a2","author":{"email":"2a07432b1b7252977a4c2dfc8e6d6d5ac763a480@netflix.com","name":"DB Tsai"},"message":"[SPARK-7279] Removed diffSum which is theoretical zero in LinearRegression and coding formating\n\nAuthor: DB Tsai <dbt@netflix.com>\n\nCloses #5809 from dbtsai/format and squashes the following commits:\n\n6904eed [DB Tsai] triger jenkins\n9146e19 [DB Tsai] initial commit","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/1c3e402e669d047410b00de9193adf3c329844a2"},{"sha":"149b3ee2dac992355adbe44e989570726c1f35d0","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":"[SPARK-7242][SQL][MLLIB] Frequent items for DataFrames\n\nFinding frequent items with possibly false positives, using the algorithm described in `http://www.cs.umd.edu/~samir/498/karp.pdf`.\npublic API under:\n```\ndf.stat.freqItems(cols: Array[String], support: Double = 0.001): DataFrame\n```\n\nThe output is a local DataFrame having the input column names with `-freqItems` appended to it. This is a single pass algorithm that may return false positives, but no false negatives.\n\ncc mengxr rxin\n\nLet's get the implementations in, I can add python API in a follow up PR.\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #5799 from brkyvz/freq-items and squashes the following commits:\n\na6ec82c [Burak Yavuz] addressed comments v?\n39b1bba [Burak Yavuz] removed toSeq\n0915e23 [Burak Yavuz] addressed comments v2.1\n3a5c177 [Burak Yavuz] addressed comments v2.0\n482e741 [Burak Yavuz] removed old import\n38e784d [Burak Yavuz] addressed comments v1.0\n8279d4d [Burak Yavuz] added default value for support\n3d82168 [Burak Yavuz] made base implementation","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/149b3ee2dac992355adbe44e989570726c1f35d0"},{"sha":"ee04413935f74b3178adbb6d8dee19b3320803e9","author":{"email":"53910baadf637a7954920bb7fe798d65a5bd4846@gmail.com","name":"rakeshchalasani"},"message":"[SPARK-7280][SQL] Add \"drop\" column/s on a data frame\n\nTakes a column name/s and returns a new DataFrame that drops a column/s.\n\nAuthor: rakeshchalasani <vnit.rakesh@gmail.com>\n\nCloses #5818 from rakeshchalasani/SPARK-7280 and squashes the following commits:\n\nce2ec09 [rakeshchalasani] Minor edit\n45c06f1 [rakeshchalasani] Change withColumnRename and format changes\nf68945a [rakeshchalasani] Minor fix\n0b9104d [rakeshchalasani] Drop one column at a time\n289afd2 [rakeshchalasani] [SPARK-7280][SQL] Add \"drop\" column/s on a data frame","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/ee04413935f74b3178adbb6d8dee19b3320803e9"},{"sha":"079733817f02c61ef814f5d9c0c8227498ff0058","author":{"email":"f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","name":"scwf"},"message":"[SPARK-7093] [SQL] Using newPredicate in NestedLoopJoin to enable code generation\n\nUsing newPredicate in NestedLoopJoin instead of InterpretedPredicate to make it can make use of code generation\n\nAuthor: scwf <wangfei1@huawei.com>\n\nCloses #5665 from scwf/NLP and squashes the following commits:\n\nd19dd31 [scwf] improvement\na887c02 [scwf] improve for NLP boundCondition","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/079733817f02c61ef814f5d9c0c8227498ff0058"},{"sha":"a0d8a61ab198b8c0ddbb3072bbe1d0e1dabc3e45","author":{"email":"f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","name":"wangfei"},"message":"[SPARK-7109] [SQL] Push down left side filter for left semi join\n\nNow in spark sql optimizer we only push down right side filter for left semi join, actually we can push down left side filter because left semi join is doing filter on left table essentially.\n\nAuthor: wangfei <wangfei1@huawei.com>\nAuthor: scwf <wangfei1@huawei.com>\n\nCloses #5677 from scwf/leftsemi and squashes the following commits:\n\n483d205 [wangfei] update with master to fix compile issue\n82df0e1 [wangfei] Merge branch 'master' of https://github.com/apache/spark into leftsemi\nd68a053 [wangfei] added apply\n8f48a3d [scwf] added test\nebadaa9 [wangfei] left filter push down for left semi join","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/a0d8a61ab198b8c0ddbb3072bbe1d0e1dabc3e45"},{"sha":"e991255e7203a0f7080efbd71f57574f46076711","author":{"email":"296ccda56873e68d94d02b8d7fd7c5480f0445e3@gmail.com","name":"Vyacheslav Baranov"},"message":"[SPARK-6913][SQL] Fixed \"java.sql.SQLException: No suitable driver found\"\n\nFixed `java.sql.SQLException: No suitable driver found` when loading DataFrame into Spark SQL if the driver is supplied with `--jars` argument.\n\nThe problem is in `java.sql.DriverManager` class that can't access drivers loaded by Spark ClassLoader.\n\nWrappers that forward requests are created for these drivers.\n\nAlso, it's not necessary any more to include JDBC drivers in `--driver-class-path` in local mode, specifying in `--jars` argument is sufficient.\n\nAuthor: Vyacheslav Baranov <slavik.baranov@gmail.com>\n\nCloses #5782 from SlavikBaranov/SPARK-6913 and squashes the following commits:\n\n510c43f [Vyacheslav Baranov] [SPARK-6913] Fixed review comments\nb2a727c [Vyacheslav Baranov] [SPARK-6913] Fixed thread race on driver registration\nc8294ae [Vyacheslav Baranov] [SPARK-6913] Fixed \"No suitable driver found\" when using using JDBC driver added with SparkContext.addJar","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/e991255e7203a0f7080efbd71f57574f46076711"},{"sha":"3ba5aaab8266822545ac82b9e733fd25cc215a77","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-5213] [SQL] Pluggable SQL Parser Support\n\nThis PR aims to make the SQL Parser Pluggable, and user can register it's own parser via Spark SQL CLI.\n\n```\n# add the jar into the classpath\n$hchengmydesktop:spark>bin/spark-sql --jars sql99.jar\n\n-- switch to \"hiveql\" dialect\n   spark-sql>SET spark.sql.dialect=hiveql;\n   spark-sql>SELECT * FROM src LIMIT 1;\n\n-- switch to \"sql\" dialect\n   spark-sql>SET spark.sql.dialect=sql;\n   spark-sql>SELECT * FROM src LIMIT 1;\n\n-- switch to a custom dialect\n   spark-sql>SET spark.sql.dialect=com.xxx.xxx.SQL99Dialect;\n   spark-sql>SELECT * FROM src LIMIT 1;\n\n-- register the non-exist SQL dialect\n   spark-sql> SET spark.sql.dialect=NotExistedClass;\n   spark-sql> SELECT * FROM src LIMIT 1;\n-- Exception will be thrown and switch to default sql dialect (\"sql\" for SQLContext and \"hiveql\" for HiveContext)\n```\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #4015 from chenghao-intel/sqlparser and squashes the following commits:\n\n493775c [Cheng Hao] update the code as feedback\n81a731f [Cheng Hao] remove the unecessary comment\naab0b0b [Cheng Hao] polish the code a little bit\n49b9d81 [Cheng Hao] shrink the comment for rebasing","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/3ba5aaab8266822545ac82b9e733fd25cc215a77"},{"sha":"473552fa5db9fa81f1a800f4ebacd23472e8c212","author":{"email":"f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","name":"scwf"},"message":"[SPARK-7123] [SQL] support table.star in sqlcontext\n\nRun following sql get error\n`SELECT r.*\nFROM testData l join testData2 r on (l.key = r.a)`\n\nAuthor: scwf <wangfei1@huawei.com>\n\nCloses #5690 from scwf/tablestar and squashes the following commits:\n\n3b2e2b6 [scwf] support table.star","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/473552fa5db9fa81f1a800f4ebacd23472e8c212"},{"sha":"beeafcfd6ee1e460c4d564cd1515d8781989b422","author":{"email":"cbb7353e6d953ef360baf960c122346276c6e320@databricks.com","name":"Patrick Wendell"},"message":"Revert \"[SPARK-5213] [SQL] Pluggable SQL Parser Support\"\n\nThis reverts commit 3ba5aaab8266822545ac82b9e733fd25cc215a77.","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/beeafcfd6ee1e460c4d564cd1515d8781989b422"},{"sha":"69a739c7f5fd002432ece203957e1458deb2f4c3","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-7282] [STREAMING] Fix the race conditions in StreamingListenerSuite\n\nFixed the following flaky test\n```Scala\n[info] StreamingListenerSuite:\n[info] - batch info reporting (782 milliseconds)\n[info] - receiver info reporting *** FAILED *** (3 seconds, 911 milliseconds)\n[info]   The code passed to eventually never returned normally. Attempted 10 times over 3.4735783689999997 seconds. Last failure message: 0 did not equal 1. (StreamingListenerSuite.scala:104)\n[info]   org.scalatest.exceptions.TestFailedDueToTimeoutException:\n[info]   at org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420)\n[info]   at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438)\n[info]   at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)\n[info]   at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:307)\n[info]   at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2.apply$mcV$sp(StreamingListenerSuite.scala:104)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2.apply(StreamingListenerSuite.scala:94)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2.apply(StreamingListenerSuite.scala:94)\n[info]   at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)\n[info]   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)\n[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)\n[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)\n[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)\n[info]   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)\n[info]   at org.scalatest.Suite$class.withFixture(Suite.scala:1122)\n[info]   at org.scalatest.FunSuite.withFixture(FunSuite.scala:1555)\n[info]   at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)\n[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)\n[info]   at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite.org$scalatest$BeforeAndAfter$$super$runTest(StreamingListenerSuite.scala:34)\n[info]   at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:200)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite.runTest(StreamingListenerSuite.scala:34)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)\n[info]   at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)\n[info]   at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)\n[info]   at scala.collection.immutable.List.foreach(List.scala:318)\n[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)\n[info]   at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)\n[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)\n[info]   at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)\n[info]   at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)\n[info]   at org.scalatest.Suite$class.run(Suite.scala:1424)\n[info]   at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)\n[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:545)\n[info]   at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite.org$scalatest$BeforeAndAfter$$super$run(StreamingListenerSuite.scala:34)\n[info]   at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:241)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite.run(StreamingListenerSuite.scala:34)\n[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:462)\n[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:671)\n[info]   at sbt.ForkMain$Run$2.call(ForkMain.java:294)\n[info]   at sbt.ForkMain$Run$2.call(ForkMain.java:284)\n[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:262)\n[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n[info]   at java.lang.Thread.run(Thread.java:745)\n[info]   Cause: org.scalatest.exceptions.TestFailedException: 0 did not equal 1\n[info]   at org.scalatest.MatchersHelper$.newTestFailedException(MatchersHelper.scala:160)\n[info]   at org.scalatest.Matchers$ShouldMethodHelper$.shouldMatcher(Matchers.scala:6231)\n[info]   at org.scalatest.Matchers$AnyShouldWrapper.should(Matchers.scala:6277)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2$$anonfun$apply$mcV$sp$1.apply$mcV$sp(StreamingListenerSuite.scala:105)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2$$anonfun$apply$mcV$sp$1.apply(StreamingListenerSuite.scala:104)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2$$anonfun$apply$mcV$sp$1.apply(StreamingListenerSuite.scala:104)\n[info]   at org.scalatest.concurrent.Eventually$class.makeAValiantAttempt$1(Eventually.scala:394)\n[info]   at org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:408)\n[info]   at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438)\n[info]   at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)\n[info]   at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:307)\n[info]   at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2.apply$mcV$sp(StreamingListenerSuite.scala:104)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2.apply(StreamingListenerSuite.scala:94)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite$$anonfun$2.apply(StreamingListenerSuite.scala:94)\n[info]   at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)\n[info]   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)\n[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)\n[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)\n[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)\n[info]   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)\n[info]   at org.scalatest.Suite$class.withFixture(Suite.scala:1122)\n[info]   at org.scalatest.FunSuite.withFixture(FunSuite.scala:1555)\n[info]   at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)\n[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)\n[info]   at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite.org$scalatest$BeforeAndAfter$$super$runTest(StreamingListenerSuite.scala:34)\n[info]   at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:200)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite.runTest(StreamingListenerSuite.scala:34)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)\n[info]   at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)\n[info]   at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)\n[info]   at scala.collection.immutable.List.foreach(List.scala:318)\n[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)\n[info]   at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)\n[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)\n[info]   at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)\n[info]   at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)\n[info]   at org.scalatest.Suite$class.run(Suite.scala:1424)\n[info]   at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)\n[info]   at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)\n[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:545)\n[info]   at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite.org$scalatest$BeforeAndAfter$$super$run(StreamingListenerSuite.scala:34)\n[info]   at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:241)\n[info]   at org.apache.spark.streaming.StreamingListenerSuite.run(StreamingListenerSuite.scala:34)\n[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:462)\n[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:671)\n[info]   at sbt.ForkMain$Run$2.call(ForkMain.java:294)\n[info]   at sbt.ForkMain$Run$2.call(ForkMain.java:284)\n[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:262)\n[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n[info]   at java.lang.Thread.run(Thread.java:745)\n```\n\nThe original codes didn't have a memory barrier in the `eventually` closure, which might fail the test, because JVM doesn't guarantee the memory consistency between different threads without  a memory barrier.\n\nThis PR used `ConcurrentLinkedQueue` to set up the memory barrier.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #5812 from zsxwing/SPARK-7282 and squashes the following commits:\n\n59115ef [zsxwing] Use SynchronizedBuffer\n014dd2b [zsxwing] Fix the race conditions in StreamingListenerSuite","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/69a739c7f5fd002432ece203957e1458deb2f4c3"},{"sha":"b5347a4664625ede6ab9d8ef6558457a34ae423f","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":" [SPARK-7248] implemented random number generators for DataFrames\n\nAdds the functions `rand` (Uniform Dist) and `randn` (Normal Dist.) as expressions to DataFrames.\n\ncc mengxr rxin\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #5819 from brkyvz/df-rng and squashes the following commits:\n\n50d69d4 [Burak Yavuz] add seed for test that failed\n4234c3a [Burak Yavuz] fix Rand expression\n13cad5c [Burak Yavuz] couple fixes\n7d53953 [Burak Yavuz] waiting for hive tests\nb453716 [Burak Yavuz] move radn with seed down\n03637f0 [Burak Yavuz] fix broken hive func\nc5909eb [Burak Yavuz] deleted old implementation of Rand\n6d43895 [Burak Yavuz] implemented random generators","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/b5347a4664625ede6ab9d8ef6558457a34ae423f"},{"sha":"36a7a6807ee58c8ba309c0e09e547919246b7f82","author":{"email":"914a64bc573f63c3e7eba683b6cec5858e463b20@gmail.com","name":"Zhan Zhang"},"message":"[SPARK-6479] [BLOCK MANAGER] Create off-heap block storage API\n\nThis is the classes for creating off-heap block storage API. It also includes the migration for Tachyon. The diff seems to be big, but it mainly just rename tachyon to offheap. New implementation for hdfs will be submit for review in spark-6112.\n\nAuthor: Zhan Zhang <zhazhan@gmail.com>\n\nCloses #5430 from zhzhan/SPARK-6479 and squashes the following commits:\n\n60acd84 [Zhan Zhang] minor change to kickoff the test\n12f54c9 [Zhan Zhang] solve merge conflicts\na54132c [Zhan Zhang] solve review comments\nffb8e00 [Zhan Zhang] rebase to sparkcontext change\n6e121e0 [Zhan Zhang] resolve review comments and restructure blockmanasger code\na7aed6c [Zhan Zhang] add Tachyon migration code\n186de31 [Zhan Zhang] initial commit for off-heap block storage api","distinct":false,"url":"https://api.github.com/repos/witgo/spark/commits/36a7a6807ee58c8ba309c0e09e547919246b7f82"}]}