{"push_id":667531909,"size":2,"distinct_size":2,"ref":"refs/heads/master","head":"6525fc0ab0c953c3346d16db0334aaf01ade7ed5","before":"fcf90b75ccf222bd2f1939addc3f8f052d2bd3ff","commits":[{"sha":"b93c97d79b42a06b48d2a8d98beccc636442541e","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-7501] [STREAMING] DAG visualization: show DStream operations\n\nThis is similar to #5999, but for streaming. Roughly 200 lines are tests.\n\nOne thing to note here is that we already do some kind of scoping thing for call sites, so this patch adds the new RDD operation scoping logic in the same place. Also, this patch adds a `try finally` block to set the relevant variables in a safer way.\n\ntdas zsxwing\n\n------------------------\n**Before**\n<img src=\"https://cloud.githubusercontent.com/assets/2133137/7625996/d88211b8-f9b4-11e4-90b9-e11baa52d6d7.png\" width=\"450px\"/>\n\n--------------------------\n**After**\n<img src=\"https://cloud.githubusercontent.com/assets/2133137/7625997/e0878f8c-f9b4-11e4-8df3-7dd611b13c87.png\" width=\"650px\"/>\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #6034 from andrewor14/dag-viz-streaming and squashes the following commits:\n\n932a64a [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming\ne685df9 [Andrew Or] Rename createRDDWith\n84d0656 [Andrew Or] Review feedback\n697c086 [Andrew Or] Fix tests\n53b9936 [Andrew Or] Set scopes for foreachRDD properly\n1881802 [Andrew Or] Refactor DStream scope names again\naf4ba8d [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming\nfd07d22 [Andrew Or] Make MQTT lower case\nf6de871 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming\n0ca1801 [Andrew Or] Remove a few unnecessary withScopes on aliases\nfa4e5fb [Andrew Or] Pass in input stream name rather than defining it from within\n1af0b0e [Andrew Or] Fix style\n074c00b [Andrew Or] Review comments\nd25a324 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming\ne4a93ac [Andrew Or] Fix tests?\n25416dc [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming\n9113183 [Andrew Or] Add tests for DStream scopes\nb3806ab [Andrew Or] Fix test\nbb80bbb [Andrew Or] Fix MIMA?\n5c30360 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming\n5703939 [Andrew Or] Rename operations that create InputDStreams\n7c4513d [Andrew Or] Group RDDs by DStream operations and batches\nbf0ab6e [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming\n05c2676 [Andrew Or] Wrap many more methods in withScope\nc121047 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming\n65ef3e9 [Andrew Or] Fix NPE\na0d3263 [Andrew Or] Scope streaming operations instead of RDD operations","distinct":true,"url":"https://api.github.com/repos/AlpineNow/spark/commits/b93c97d79b42a06b48d2a8d98beccc636442541e"},{"sha":"6525fc0ab0c953c3346d16db0334aaf01ade7ed5","author":{"email":"081ca5f8e359b969a806e4ba1eafbafb637fc7cb@gmail.com","name":"Jihong MA"},"message":"[SPARK-7063] when lz4 compression is used, it causes core dump\n\nthis fix is to solve one issue found in lz4 1.2.0, which caused core dump in Spark Core with IBM JDK.  that issue is fixed in lz4 1.3.0 version.\n\nAuthor: Jihong MA <linlin200605@gmail.com>\n\nCloses #6226 from JihongMA/SPARK-7063-1 and squashes the following commits:\n\n0cca781 [Jihong MA] SPARK-7063\n4559ed5 [Jihong MA] SPARK-7063\ndaa520f [Jihong MA] SPARK-7063 upgrade lz4 jars\n71738ee [Jihong MA] Merge remote-tracking branch 'upstream/master'\ndfaa971 [Jihong MA] SPARK-7265 minor fix of the content\nace454d [Jihong MA] SPARK-7265 take out PySpark on YARN limitation\n9ea0832 [Jihong MA] Merge remote-tracking branch 'upstream/master'\nd5bf3f5 [Jihong MA] Merge remote-tracking branch 'upstream/master'\n7b842e6 [Jihong MA] Merge remote-tracking branch 'upstream/master'\n9c84695 [Jihong MA] SPARK-7265 address review comment\na399aa6 [Jihong MA] SPARK-7265 Improving documentation for Spark SQL Hive support","distinct":true,"url":"https://api.github.com/repos/AlpineNow/spark/commits/6525fc0ab0c953c3346d16db0334aaf01ade7ed5"}]}