{"push_id":633067654,"size":479,"distinct_size":46,"ref":"refs/heads/direct-kafka-python-api","head":"9da49be0cf2e569a9c871dd7bbb3aee7820f9e0e","before":"c3019515e3636568db6e625705b774f792c8bc2b","commits":[{"sha":"8c468a6600e0deb5464990df60148212e64fdecd","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PID_DIR when running Thrift server test suites\n\nThis is a follow-up of #4720. By default, `spark-daemon.sh` writes PID files under `/tmp`, which makes it impossible to start multiple server instances simultaneously. This PR sets `SPARK_PID_DIR` to Spark home directory to workaround this problem.\n\nMany thanks to chenghao-intel for pointing out this issue!\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/apache/spark/4758)\n<!-- Reviewable:end -->\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #4758 from liancheng/thriftserver-pid-dir and squashes the following commits:\n\n252fa0f [Cheng Lian] Uses temporary directory as Thrift server PID directory\n1b3d1e3 [Cheng Lian] Sets SPARK_HOME as SPARK_PID_DIR when running Thrift server test suites","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/8c468a6600e0deb5464990df60148212e64fdecd"},{"sha":"e0e64ba4b1b8eb72e856286f756c65fa22ab0a36","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-6055] [PySpark] fix incorrect __eq__ of DataType\n\nThe _eq_ of DataType is not correct, class cache is not use correctly (created class can not be find by dataType), then it will create lots of classes (saved in _cached_cls), never released.\n\nAlso, all same DataType have same hash code, there will be many object in a dict with the same hash code, end with hash attach, it's very slow to access this dict (depends on the implementation of CPython).\n\nThis PR also improve the performance of inferSchema (avoid the unnecessary converter of object).\n\ncc pwendell  JoshRosen\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #4808 from davies/leak and squashes the following commits:\n\n6a322a4 [Davies Liu] tests refactor\n3da44fc [Davies Liu] fix __eq__ of Singleton\n534ac90 [Davies Liu] add more checks\n46999dc [Davies Liu] fix tests\nd9ae973 [Davies Liu] fix memory leak in sql","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/e0e64ba4b1b8eb72e856286f756c65fa22ab0a36"},{"sha":"dba08d1fc3bdb9245aefe695970354df088a93b6","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-6070] [yarn] Remove unneeded classes from shuffle service jar.\n\nThese may conflict with the classes already in the NM. We shouldn't\nbe repackaging them.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #4820 from vanzin/SPARK-6070 and squashes the following commits:\n\n871b566 [Marcelo Vanzin] The \"d'oh how didn't I think of it before\" solution.\n3cba946 [Marcelo Vanzin] Use profile instead, so that dependencies don't need to be explicitly listed.\n7a18a1b [Marcelo Vanzin] [SPARK-6070] [yarn] Remove unneeded classes from shuffle service jar.","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/dba08d1fc3bdb9245aefe695970354df088a93b6"},{"sha":"6d8e5fbc0d83411174ffa59ff6a761a862eca32c","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":"[SPARK-5979][SPARK-6032] Smaller safer --packages fix\n\npwendell tdas\nThis is the safer parts of PR #4754:\n - SPARK-5979: All dependencies with the groupId `org.apache.spark` passed through `--packages`, were being excluded from the dependency tree on the assumption that they would be in the assembly jar. This is not the case, therefore the exclusion rules had to be defined more explicitly.\n - SPARK-6032: Ivy prints a whole lot of logs while retrieving dependencies. These were printed to `System.out`. Moved the logging to `System.err`.\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #4802 from brkyvz/simple-streaming-fix and squashes the following commits:\n\ne0f38cb [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into simple-streaming-fix\nbad921c [Burak Yavuz] [SPARK-5979][SPARK-6032] Smaller safer fix","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/6d8e5fbc0d83411174ffa59ff6a761a862eca32c"},{"sha":"9168259813713a12251fb0d457ffbbed8ba857f8","author":{"email":"cbb7353e6d953ef360baf960c122346276c6e320@databricks.com","name":"Patrick Wendell"},"message":"MAINTENANCE: Automated closing of pull requests.\n\nThis commit exists to close the following pull requests on Github:\n\nCloses #1128 (close requested by 'srowen')\nCloses #3425 (close requested by 'srowen')\nCloses #4770 (close requested by 'srowen')\nCloses #2813 (close requested by 'srowen')","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/9168259813713a12251fb0d457ffbbed8ba857f8"},{"sha":"e6003f0a571ba44fcd011e695c8622e11cfee7dd","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-5775] [SQL] BugFix: GenericRow cannot be cast to SpecificMutableRow when nested data and partitioned table\n\nThis PR adapts anselmevignon's #4697 to master and branch-1.3. Please refer to PR description of #4697 for details.\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/apache/spark/4792)\n<!-- Reviewable:end -->\n\nAuthor: Cheng Lian <lian@databricks.com>\nAuthor: Cheng Lian <liancheng@users.noreply.github.com>\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #4792 from liancheng/spark-5775 and squashes the following commits:\n\n538f506 [Cheng Lian] Addresses comments\ncee55cf [Cheng Lian] Merge pull request #4 from yhuai/spark-5775-yin\nb0b74fb [Yin Huai] Remove runtime pattern matching.\nca6e038 [Cheng Lian] Fixes SPARK-5775","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/e6003f0a571ba44fcd011e695c8622e11cfee7dd"},{"sha":"b36b1bc22ea73669b0f69ed21e77d47fb0a7cd5d","author":{"email":"4e21f4fd0c745866d3c39b83ddb890f7ef0b129b@gmail.com","name":"Michael Griffiths"},"message":"SPARK-6063 MLlib doesn't pass mvn scalastyle check due to UTF chars in LDAModel.scala\n\nRemove unicode characters from MLlib file.\n\nAuthor: Michael Griffiths <msjgriffiths@gmail.com>\nAuthor: Griffiths, Michael (NYC-RPM) <michael.griffiths@reprisemedia.com>\n\nCloses #4815 from msjgriffiths/SPARK-6063 and squashes the following commits:\n\nbcd7de1 [Griffiths, Michael (NYC-RPM)] Change \\u201D quote marks around 'theta' to standard single apostrophe (\\x27)\n38eb535 [Michael Griffiths] Merge pull request #2 from apache/master\nb08e865 [Michael Griffiths] Merge pull request #1 from apache/master","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/b36b1bc22ea73669b0f69ed21e77d47fb0a7cd5d"},{"sha":"f91298e2c597e45af461931919372da5d33ae3da","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"SPARK-5983 [WEBUI] Don't respond to HTTP TRACE in HTTP-based UIs\n\nDisallow TRACE HTTP method in servlets\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #4765 from srowen/SPARK-5983 and squashes the following commits:\n\n421b25b [Sean Owen] Disallow TRACE HTTP method in servlets","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/f91298e2c597e45af461931919372da5d33ae3da"},{"sha":"86fcdaef62dbe624233e364ffe43fe3a1da893f0","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"SPARK-1965 [WEBUI] Spark UI throws NPE on trying to load the app page for non-existent app\n\nDon't throw NPE if appId is unknown. kayousterhout is this a decent enough band-aid for avoiding a full-blown NPE? it should just render empty content instead\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #4777 from srowen/SPARK-1965 and squashes the following commits:\n\n7e16590 [Sean Owen] Update app not found message\ncb878d6 [Sean Owen] Return basic \"not found\" page for unknown appId\nd8270da [Sean Owen] Don't throw NPE if appId is unknown","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/86fcdaef62dbe624233e364ffe43fe3a1da893f0"},{"sha":"643300a6e27dac3822f9a3ced0ad5fb3b4f2ad75","author":{"email":"08777c6567f7efed43a671c02810951804f92197@gmail.com","name":"Evan Yu"},"message":"SPARK-5984: Fix TimSort bug causes ArrayOutOfBoundsException\n\nFix TimSort bug which causes a ArrayOutOfBoundsException.\n\nUsing the proposed fix here\nhttp://envisage-project.eu/proving-android-java-and-python-sorting-algorithm-is-broken-and-how-to-fix-it/\n\nAuthor: Evan Yu <ehotou@gmail.com>\n\nCloses #4804 from hotou/SPARK-5984 and squashes the following commits:\n\n3421b6c [Evan Yu] SPARK-5984: Add info to LICENSE\ne61c6b8 [Evan Yu] SPARK-5984: Fix license and document\n6ccc280 [Evan Yu] SPARK-5984: Add License header to file\ne06c0d2 [Evan Yu] SPARK-5984: Add License header to file\n4d95f75 [Evan Yu] SPARK-5984: Fix TimSort bug causes ArrayOutOfBoundsException\n479a106 [Evan Yu] SPARK-5984: Fix TimSort bug causes ArrayOutOfBoundsException","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/643300a6e27dac3822f9a3ced0ad5fb3b4f2ad75"},{"sha":"2df5f1f00661cd31b9fc37e80345a3f5f856c95f","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-6075] Fix bug in that caused lost accumulator updates: do not store WeakReferences in localAccums map\n\nThis fixes a non-deterministic bug introduced in #4021 that could cause tasks' accumulator updates to be lost.  The problem is that `localAccums` should not hold weak references: after the task finishes running there won't be any strong references to these local accumulators, so they can get garbage-collected before the executor reads the `localAccums` map.  We don't need weak references here anyways, since this map is cleared at the end of each task.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #4835 from JoshRosen/SPARK-6075 and squashes the following commits:\n\n4f4b5b2 [Josh Rosen] Remove defensive assertions that caused test failures in code unrelated to this change\n120c7b0 [Josh Rosen] [SPARK-6075] Do not store WeakReferences in localAccums map","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/2df5f1f00661cd31b9fc37e80345a3f5f856c95f"},{"sha":"fd8d283eeb98e310b1e85ef8c3a8af9e547ab5e0","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-6074] [sql] Package pyspark sql bindings.\n\nThis is needed for the SQL bindings to work on Yarn.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #4822 from vanzin/SPARK-6074 and squashes the following commits:\n\nfb52001 [Marcelo Vanzin] [SPARK-6074] [sql] Package pyspark sql bindings.","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/fd8d283eeb98e310b1e85ef8c3a8af9e547ab5e0"},{"sha":"aedbbaa3dda9cbc154cd52c07f6d296b972b0eb2","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-6053][MLLIB] support save/load in PySpark's ALS\n\nA simple wrapper to save/load `MatrixFactorizationModel` in Python. jkbradley\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #4811 from mengxr/SPARK-5991 and squashes the following commits:\n\nf135dac [Xiangrui Meng] update save doc\n57e5200 [Xiangrui Meng] address comments\n06140a4 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into SPARK-5991\n282ec8d [Xiangrui Meng] support save/load in PySpark's ALS","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/aedbbaa3dda9cbc154cd52c07f6d296b972b0eb2"},{"sha":"3f00bb3ef1384fabf86a68180d40a1a515f6f5e3","author":{"email":"8757c4a255384de9fc2175f17a84f948e7081d30@gmail.com","name":"MechCoder"},"message":"[SPARK-6083] [MLLib] [DOC] Make Python API example consistent in NaiveBayes\n\nAuthor: MechCoder <manojkumarsivaraj334@gmail.com>\n\nCloses #4834 from MechCoder/spark-6083 and squashes the following commits:\n\n1cdd7b5 [MechCoder] Add parse function\n65bbbe9 [MechCoder] [SPARK-6083] Make Python API example consistent in NaiveBayes","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/3f00bb3ef1384fabf86a68180d40a1a515f6f5e3"},{"sha":"d8fb40edea7c8c811814f1ff288d59178928964b","author":{"email":"79478dbbc8641d1d6545f348f655464aefbda99b@intel.com","name":"Saisai Shao"},"message":"[Streaming][Minor]Fix some error docs in streaming examples\n\nSmall changes, please help to review, thanks a lot.\n\nAuthor: Saisai Shao <saisai.shao@intel.com>\n\nCloses #4837 from jerryshao/doc-fix and squashes the following commits:\n\n545291a [Saisai Shao] Fix some error docs in streaming examples","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/d8fb40edea7c8c811814f1ff288d59178928964b"},{"sha":"948c2390ab004ad5cf3876d87c05d3e43a9aa9e0","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"SPARK-3357 [CORE] Internal log messages should be set at DEBUG level instead of INFO\n\nDemote some 'noisy' log messages to debug level. I added a few more, to include everything that gets logged in stanzas like this:\n\n```\n15/03/01 00:03:54 INFO BlockManager: Removing broadcast 0\n15/03/01 00:03:54 INFO BlockManager: Removing block broadcast_0_piece0\n15/03/01 00:03:54 INFO MemoryStore: Block broadcast_0_piece0 of size 839 dropped from memory (free 277976091)\n15/03/01 00:03:54 INFO BlockManagerInfo: Removed broadcast_0_piece0 on localhost:49524 in memory (size: 839.0 B, free: 265.1 MB)\n15/03/01 00:03:54 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0\n15/03/01 00:03:54 INFO BlockManager: Removing block broadcast_0\n15/03/01 00:03:54 INFO MemoryStore: Block broadcast_0 of size 1088 dropped from memory (free 277977179)\n15/03/01 00:03:54 INFO ContextCleaner: Cleaned broadcast 0\n```\n\nas well as regular messages like\n\n```\n15/03/01 00:02:33 INFO MemoryStore: ensureFreeSpace(2640) called with curMem=47322, maxMem=278019440\n```\n\nWDYT? good or should some be left alone?\n\nCC mengxr who suggested some of this.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #4838 from srowen/SPARK-3357 and squashes the following commits:\n\ndce75c1 [Sean Owen] Back out some debug level changes\nd9b784d [Sean Owen] Demote some 'noisy' log messages to debug level","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/948c2390ab004ad5cf3876d87c05d3e43a9aa9e0"},{"sha":"49c7a8f6f33d64d7e6c35391f83121440844a41d","author":{"email":"0da04896db04dd2393029a4b0c77df7858a20dea@gmail.com","name":"Lianhui Wang"},"message":"[SPARK-6103][Graphx]remove unused class to import in EdgeRDDImpl\n\nClass TaskContext is unused in EdgeRDDImpl, so we need to remove it from import list.\n\nAuthor: Lianhui Wang <lianhuiwang09@gmail.com>\n\nCloses #4846 from lianhuiwang/SPARK-6103 and squashes the following commits:\n\n31aed64 [Lianhui Wang] remove unused class to import in EdgeRDDImpl","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/49c7a8f6f33d64d7e6c35391f83121440844a41d"},{"sha":"39a54b40aff66816f8b8f5c6133eaaad6eaecae1","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-6073][SQL] Need to refresh metastore cache after append data in CreateMetastoreDataSourceAsSelect\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-6073\n\nliancheng\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #4824 from yhuai/refreshCache and squashes the following commits:\n\nb9542ef [Yin Huai] Refresh metadata cache in the Catalog in CreateMetastoreDataSourceAsSelect.","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/39a54b40aff66816f8b8f5c6133eaaad6eaecae1"},{"sha":"3efd8bb6cf139ce094ff631c7a9c1eb93fdcd566","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-6052][SQL]In JSON schema inference, we should always set containsNull of an ArrayType to true\n\nAlways set `containsNull = true` when infer the schema of JSON datasets. If we set `containsNull` based on records we scanned, we may miss arrays with null values when we do sampling. Also, because future data can have arrays with null values, if we convert JSON data to parquet, always setting `containsNull = true` is a more robust way to go.\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-6052\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #4806 from yhuai/jsonArrayContainsNull and squashes the following commits:\n\n05eab9d [Yin Huai] Change containsNull to true.","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/3efd8bb6cf139ce094ff631c7a9c1eb93fdcd566"},{"sha":"95ac68bf127b5370c13d6bc15adbda78228829cc","author":{"email":"1fc3ade0ca66a5997ee9eee1a4074efe89a462b1@us.ibm.com","name":"Kenneth Myers"},"message":"[SPARK-6111] Fixed usage string in documentation.\n\nUsage info in documentation does not match actual usage info.\n\nDoc string usage says ```Usage: network_wordcount.py <zk> <topic>``` whereas the actual usage is ```Usage: kafka_wordcount.py <zk> <topic>```\n\nAuthor: Kenneth Myers <myerske@us.ibm.com>\n\nCloses #4852 from kennethmyers/kafka_wordcount_documentation_fix and squashes the following commits:\n\n3855325 [Kenneth Myers] Fixed usage string in documentation.","distinct":false,"url":"https://api.github.com/repos/jerryshao/apache-spark/commits/95ac68bf127b5370c13d6bc15adbda78228829cc"}]}