{"push_id":640304504,"size":64,"distinct_size":20,"ref":"refs/heads/viz","head":"5e22946945f683927cabafeb0ede3bc8e275e4a0","before":"6a7cdcaed6bb6fd856bd7e2e15b0d78cbdb0b2d1","commits":[{"sha":"04e44b37cc04f62fbf9e08c7076349e0a4d12ea8","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-4897] [PySpark] Python 3 support\n\nThis PR update PySpark to support Python 3 (tested with 3.4).\n\nKnown issue: unpickle array from Pyrolite is broken in Python 3, those tests are skipped.\n\nTODO: ec2/spark-ec2.py is not fully tested with python3.\n\nAuthor: Davies Liu <davies@databricks.com>\nAuthor: twneale <twneale@gmail.com>\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #5173 from davies/python3 and squashes the following commits:\n\nd7d6323 [Davies Liu] fix tests\n6c52a98 [Davies Liu] fix mllib test\n99e334f [Davies Liu] update timeout\nb716610 [Davies Liu] Merge branch 'master' of github.com:apache/spark into python3\ncafd5ec [Davies Liu] adddress comments from @mengxr\nbf225d7 [Davies Liu] Merge branch 'master' of github.com:apache/spark into python3\n179fc8d [Davies Liu] tuning flaky tests\n8c8b957 [Davies Liu] fix ResourceWarning in Python 3\n5c57c95 [Davies Liu] Merge branch 'master' of github.com:apache/spark into python3\n4006829 [Davies Liu] fix test\n2fc0066 [Davies Liu] add python3 path\n71535e9 [Davies Liu] fix xrange and divide\n5a55ab4 [Davies Liu] Merge branch 'master' of github.com:apache/spark into python3\n125f12c [Davies Liu] Merge branch 'master' of github.com:apache/spark into python3\ned498c8 [Davies Liu] fix compatibility with python 3\n820e649 [Davies Liu] Merge branch 'master' of github.com:apache/spark into python3\ne8ce8c9 [Davies Liu] Merge branch 'master' of github.com:apache/spark into python3\nad7c374 [Davies Liu] fix mllib test and warning\nef1fc2f [Davies Liu] fix tests\n4eee14a [Davies Liu] Merge branch 'master' of github.com:apache/spark into python3\n20112ff [Davies Liu] Merge branch 'master' of github.com:apache/spark into python3\n59bb492 [Davies Liu] fix tests\n1da268c [Davies Liu] Merge branch 'master' of github.com:apache/spark into python3\nca0fdd3 [Davies Liu] fix code style\n9563a15 [Davies Liu] add imap back for python 2\n0b1ec04 [Davies Liu] make python examples work with Python 3\nd2fd566 [Davies Liu] Merge branch 'master' of github.com:apache/spark into python3\na716d34 [Davies Liu] test with python 3.4\nf1700e8 [Davies Liu] fix test in python3\n671b1db [Davies Liu] fix test in python3\n692ff47 [Davies Liu] fix flaky test\n7b9699f [Davies Liu] invalidate import cache for Python 3.3+\n9c58497 [Davies Liu] fix kill worker\n309bfbf [Davies Liu] keep compatibility\n5707476 [Davies Liu] cleanup, fix hash of string in 3.3+\n8662d5b [Davies Liu] Merge branch 'master' of github.com:apache/spark into python3\nf53e1f0 [Davies Liu] fix tests\n70b6b73 [Davies Liu] compile ec2/spark_ec2.py in python 3\na39167e [Davies Liu] support customize class in __main__\n814c77b [Davies Liu] run unittests with python 3\n7f4476e [Davies Liu] mllib tests passed\nd737924 [Davies Liu] pass ml tests\n375ea17 [Davies Liu] SQL tests pass\n6cc42a9 [Davies Liu] rename\n431a8de [Davies Liu] streaming tests pass\n78901a7 [Davies Liu] fix hash of serializer in Python 3\n24b2f2e [Davies Liu] pass all RDD tests\n35f48fe [Davies Liu] run future again\n1eebac2 [Davies Liu] fix conflict in ec2/spark_ec2.py\n6e3c21d [Davies Liu] make cloudpickle work with Python3\n2fb2db3 [Josh Rosen] Guard more changes behind sys.version; still doesn't run\n1aa5e8f [twneale] Turned out `pickle.DictionaryType is dict` == True, so swapped it out\n7354371 [twneale] buffer --> memoryview  I'm not super sure if this a valid change, but the 2.7 docs recommend using memoryview over buffer where possible, so hoping it'll work.\nb69ccdf [twneale] Uses the pure python pickle._Pickler instead of c-extension _pickle.Pickler. It appears pyspark 2.7 uses the pure python pickler as well, so this shouldn't degrade pickling performance (?).\nf40d925 [twneale] xrange --> range\ne104215 [twneale] Replaces 2.7 types.InstsanceType with 3.4 `object`....could be horribly wrong depending on how types.InstanceType is used elsewhere in the package--see http://bugs.python.org/issue8206\n79de9d0 [twneale] Replaces python2.7 `file` with 3.4 _io.TextIOWrapper\n2adb42d [Josh Rosen] Fix up some import differences between Python 2 and 3\n854be27 [Josh Rosen] Run `futurize` on Python code:\n7c5b4ce [Josh Rosen] Remove Python 3 check in shell.py.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/04e44b37cc04f62fbf9e08c7076349e0a4d12ea8"},{"sha":"5fe43433529346788e8c343d338a5b7dc169cf58","author":{"email":"c95969b69933e0f49af50a76ac2e9a9cf10c0740@alibaba-inc.com","name":"云峤"},"message":"SPARK-6927 [SQL] Sorting Error when codegen on\n\nFix this error by adding BinaryType comparor in GenerateOrdering.\nJIRA https://issues.apache.org/jira/browse/SPARK-6927\n\nAuthor: 云峤 <chensong.cs@alibaba-inc.com>\n\nCloses #5524 from kaka1992/fix-codegen-sort and squashes the following commits:\n\nd7e2afe [云峤] fix codegen sorting error","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/5fe43433529346788e8c343d338a5b7dc169cf58"},{"sha":"6183b5e2caedd074073d0f6cb6609a634e2f5194","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-6911] [SQL] improve accessor for nested types\n\nSupport access columns by index in Python:\n```\n>>> df[df[0] > 3].collect()\n[Row(age=5, name=u'Bob')]\n```\n\nAccess items in ArrayType or MapType\n```\n>>> df.select(df.l.getItem(0), df.d.getItem(\"key\")).show()\n>>> df.select(df.l[0], df.d[\"key\"]).show()\n```\n\nAccess field in StructType\n```\n>>> df.select(df.r.getField(\"b\")).show()\n>>> df.select(df.r.a).show()\n```\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #5513 from davies/access and squashes the following commits:\n\ne04d5a0 [Davies Liu] Update run-tests-jenkins\n7ada9eb [Davies Liu] update timeout\nd125ac4 [Davies Liu] check column name, improve scala tests\n6b62540 [Davies Liu] fix test\ndb15b42 [Davies Liu] Merge branch 'master' of github.com:apache/spark into access\n6c32e79 [Davies Liu] add scala tests\n11f1df3 [Davies Liu] improve accessor for nested types","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/6183b5e2caedd074073d0f6cb6609a634e2f5194"},{"sha":"d96608674f6c2ff3abb13c65d80c1a3872206710","author":{"email":"f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","name":"scwf"},"message":"[SQL][Minor] Fix foreachUp of treenode\n\n`foreachUp` should runs the given function recursively on [[children]] then on this node(just like transformUp). The current implementation does not follow this.\n\nThis will leads to checkanalysis do not check from bottom of logical tree.\n\nAuthor: scwf <wangfei1@huawei.com>\nAuthor: Fei Wang <wangfei1@huawei.com>\n\nCloses #5518 from scwf/patch-1 and squashes the following commits:\n\n18e28b2 [scwf] added a test case\n1ccbfa8 [Fei Wang] fix foreachUp","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/d96608674f6c2ff3abb13c65d80c1a3872206710"},{"sha":"1e43851d6455f65b850ea0327d0e92f65395d23f","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-6899][SQL] Fix type mismatch when using codegen with Average on DecimalType\n\nJIRA https://issues.apache.org/jira/browse/SPARK-6899\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #5517 from viirya/fix_codegen_average and squashes the following commits:\n\n8ae5f65 [Liang-Chi Hsieh] Add the case of DecimalType.Unlimited to Average.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/1e43851d6455f65b850ea0327d0e92f65395d23f"},{"sha":"e5949c287ed19e78b6eecc61c3e88a07ad452eb9","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-6966][SQL] Use correct ClassLoader for JDBC Driver\n\nOtherwise we cannot add jars with drivers after the fact.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #5543 from marmbrus/jdbcClassloader and squashes the following commits:\n\nd9930f3 [Michael Armbrust] fix imports\n73d0614 [Michael Armbrust] [SPARK-6966][SQL] Use correct ClassLoader for JDBC Driver","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/e5949c287ed19e78b6eecc61c3e88a07ad452eb9"},{"sha":"8220d5265f1bbea9dfdaeec4f2d06d7fe24c0bc3","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-6972][SQL] Add Coalesce to DataFrame\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #5545 from marmbrus/addCoalesce and squashes the following commits:\n\n9fdf3f6 [Michael Armbrust] [SPARK-6972][SQL] Add Coalesce to DataFrame","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/8220d5265f1bbea9dfdaeec4f2d06d7fe24c0bc3"},{"sha":"f7a25644ed5b3b49fe7f33743bec3d95cdf7913e","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"SPARK-6846 [WEBUI] Stage kill URL easy to accidentally trigger and possibility for security issue\n\nkill endpoints now only accept a POST (kill stage, master kill app, master kill driver); kill link now POSTs\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #5528 from srowen/SPARK-6846 and squashes the following commits:\n\n137ac9f [Sean Owen] Oops, fix scalastyle line length probelm\n7c5f961 [Sean Owen] Add Imran's test of kill link\n59f447d [Sean Owen] kill endpoints now only accept a POST (kill stage, master kill app, master kill driver); kill link now POSTs","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/f7a25644ed5b3b49fe7f33743bec3d95cdf7913e"},{"sha":"4527761bcd6501c362baf2780905a0018b9a74ba","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":" [SPARK-6046] [core] Reorganize deprecated config support in SparkConf.\n\nThis change tries to follow the chosen way for handling deprecated\nconfigs in SparkConf: all values (old and new) are kept in the conf\nobject, and newer names take precedence over older ones when\nretrieving the value.\n\nWarnings are logged when config options are set, which generally happens\non the driver node (where the logs are most visible).\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #5514 from vanzin/SPARK-6046 and squashes the following commits:\n\n9371529 [Marcelo Vanzin] Avoid math.\n6cf3f11 [Marcelo Vanzin] Review feedback.\n2445d48 [Marcelo Vanzin] Fix (and cleanup) update interval initialization.\nb6824be [Marcelo Vanzin] Clean up the other deprecated config use also.\nab20351 [Marcelo Vanzin] Update FsHistoryProvider to only retrieve new config key.\n2c93209 [Marcelo Vanzin] [SPARK-6046] [core] Reorganize deprecated config support in SparkConf.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/4527761bcd6501c362baf2780905a0018b9a74ba"},{"sha":"f6a9a57a72767f48fcc02e5fda4d6eafa67aebde","author":{"email":"29e05320a613faa71033b8bf4c133bf8ae9edea8@palantir.com","name":"Punya Biswal"},"message":"[SPARK-6952] Handle long args when detecting PID reuse\n\nsbin/spark-daemon.sh used\n\n    ps -p \"$TARGET_PID\" -o args=\n\nto figure out whether the process running with the expected PID is actually a Spark\ndaemon. When running with a large classpath, the output of ps gets\ntruncated and the check fails spuriously.\n\nThis weakens the check to see if it's a java command (which is something\nwe do in other parts of the script) rather than looking for the specific\nmain class name. This means that SPARK-4832 might happen under a\nslightly broader range of circumstances (a java program happened to\nreuse the same PID), but it seems worthwhile compared to failing\nconsistently with a large classpath.\n\nAuthor: Punya Biswal <pbiswal@palantir.com>\n\nCloses #5535 from punya/feature/SPARK-6952 and squashes the following commits:\n\n7ea12d1 [Punya Biswal] Handle long args when detecting PID reuse","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/f6a9a57a72767f48fcc02e5fda4d6eafa67aebde"},{"sha":"dc48ba9f9f7449dd2f12cbad288b65c8119d9284","author":{"email":"c40433a0b9f1df8d6cfdcc84006d6c506907d641@huawei.com","name":"linweizhong"},"message":"[SPARK-6604][PySpark]Specify ip of python server scoket\n\nIn driver now will start a server socket and use a wildcard ip, use 127.0.0.0 is more reasonable, as we only use it by local Python process.\n/cc davies\n\nAuthor: linweizhong <linweizhong@huawei.com>\n\nCloses #5256 from Sephiroth-Lin/SPARK-6604 and squashes the following commits:\n\n7b3c633 [linweizhong] rephrase","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/dc48ba9f9f7449dd2f12cbad288b65c8119d9284"},{"sha":"c84d91692aa25c01882bcc3f9fd5de3cfa786195","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-6957] [SPARK-6958] [SQL] improve API compatibility to pandas\n\n```\nselect(['cola', 'colb'])\n\ngroupby(['colA', 'colB'])\ngroupby([df.colA, df.colB])\n\ndf.sort('A', ascending=True)\ndf.sort(['A', 'B'], ascending=True)\ndf.sort(['A', 'B'], ascending=[1, 0])\n```\n\ncc rxin\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #5544 from davies/compatibility and squashes the following commits:\n\n4944058 [Davies Liu] add docstrings\nadb2816 [Davies Liu] Merge branch 'master' of github.com:apache/spark into compatibility\nbcbbcab [Davies Liu] support ascending as list\n8dabdf0 [Davies Liu] improve API compatibility to pandas","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/c84d91692aa25c01882bcc3f9fd5de3cfa786195"},{"sha":"50ab8a6543ad5c31e89c16df374d0cb13222fd1e","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-2669] [yarn] Distribute client configuration to AM.\n\nCurrently, when Spark launches the Yarn AM, the process will use\nthe local Hadoop configuration on the node where the AM launches,\nif one is present. A more correct approach is to use the same\nconfiguration used to launch the Spark job, since the user may\nhave made modifications (such as adding app-specific configs).\n\nThe approach taken here is to use the distributed cache to make\nall files in the Hadoop configuration directory available to the\nAM. This is a little overkill since only the AM needs them (the\nexecutors use the broadcast Hadoop configuration from the driver),\nbut is the easier approach.\n\nEven though only a few files in that directory may end up being\nused, all of them are uploaded. This allows supporting use cases\nsuch as when auxiliary configuration files are used for SSL\nconfiguration, or when uploading a Hive configuration directory.\nNot all of these may be reflected in a o.a.h.conf.Configuration object,\nbut may be needed when a driver in cluster mode instantiates, for\nexample, a HiveConf object instead.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #4142 from vanzin/SPARK-2669 and squashes the following commits:\n\nf5434b9 [Marcelo Vanzin] Merge branch 'master' into SPARK-2669\n013f0fb [Marcelo Vanzin] Review feedback.\nf693152 [Marcelo Vanzin] Le sigh.\ned45b7d [Marcelo Vanzin] Zip all config files and upload them as an archive.\n5927b6b [Marcelo Vanzin] Merge branch 'master' into SPARK-2669\ncbb9fb3 [Marcelo Vanzin] Remove stale test.\ne3e58d0 [Marcelo Vanzin] Merge branch 'master' into SPARK-2669\ne3d0613 [Marcelo Vanzin] Review feedback.\n34bdbd8 [Marcelo Vanzin] Fix test.\n022a688 [Marcelo Vanzin] Merge branch 'master' into SPARK-2669\na77ddd5 [Marcelo Vanzin] Merge branch 'master' into SPARK-2669\n79221c7 [Marcelo Vanzin] [SPARK-2669] [yarn] Distribute client configuration to AM.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/50ab8a6543ad5c31e89c16df374d0cb13222fd1e"},{"sha":"a83571acc938582865efb41645aa1e414f339e46","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-6113] [ml] Stabilize DecisionTree API\n\nThis is a PR for cleaning up and finalizing the DecisionTree API.  PRs for ensembles will follow once this is merged.\n\n### Goal\n\nHere is the description copied from the JIRA (for both trees and ensembles):\n\n> **Issue**: The APIs for DecisionTree and ensembles (RandomForests and GradientBoostedTrees) have been experimental for a long time. The API has become very convoluted because trees and ensembles have many, many variants, some of which we have added incrementally without a long-term design.\n> **Proposal**: This JIRA is for discussing changes required to finalize the APIs. After we discuss, I will make a PR to update the APIs and make them non-Experimental. This will require making many breaking changes; see the design doc for details.\n> **[Design doc](https://docs.google.com/document/d/1rJ_DZinyDG3PkYkAKSsQlY0QgCeefn4hUv7GsPkzBP4)** : This outlines current issues and the proposed API.\n\nOverall code layout:\n* The old API in mllib.tree.* will remain the same.\n* The new API will reside in ml.classification.* and ml.regression.*\n\n### Summary of changes\n\nOld API\n* Exactly the same, except I made 1 method in Loss private (but that is not a breaking change since that method was introduced after the Spark 1.3 release).\n\nNew APIs\n* Under Pipeline API\n* The new API preserves functionality, except:\n  * New API does NOT store prob (probability of label in classification).  I want to have it store the full vector of probabilities but feel that should be in a later PR.\n* Use abstractions for parameters, estimators, and models to avoid code duplication\n* Limit parameters to relevant algorithms\n* For enum-like types, only expose Strings\n  * We can make these pluggable later on by adding new parameters.  That is a far-future item.\n\nTest suites\n* I organized DecisionTreeSuite, but I made absolutely no changes to the tests themselves.\n* The test suites for the new API only test (a) similarity with the results of the old API and (b) elements of the new API.\n  * After code is moved to this new API, we should move the tests from the old suites which test the internals.\n\n### Details\n\n#### Changed names\n\nParameters\n* useNodeIdCache -> cacheNodeIds\n\n#### Other changes\n\n* Split: Changed categories to set instead of list\n\n#### Non-decision tree changes\n* AttributeGroup\n  * Added parentheses to toMetadata, toStructField methods (These were removed in a previous PR, but I ran into 1 issue with the Scala compiler not being able to disambiguate between a toMetadata method with no parentheses and a toMetadata method which takes 1 argument.)\n* Attributes\n  * Renamed: toMetadata -> toMetadataImpl\n  * Added toMetadata methods which return ML metadata (keyed with “ML_ATTR”)\n  * NominalAttribute: Added getNumValues method which examines both numValues and values.\n* Params.inheritValues: Checks whether the parent param really belongs to the child (to allow Estimator-Model pairs with different sets of parameters)\n\n### Questions for reviewers\n\n* Is \"DecisionTreeClassificationModel\" too long a name?\n* Is this OK in the docs?\n```\nclass DecisionTreeRegressor extends TreeRegressor[DecisionTreeRegressionModel] with DecisionTreeParams[DecisionTreeRegressor] with TreeRegressorParams[DecisionTreeRegressor]\n```\n\n### Future\n\nWe should open up the abstractions at some point.  E.g., it would be useful to be able to set tree-related parameters in 1 place and then pass those to multiple tree-based algorithms.\n\nFollow-up JIRAs will be (in this order):\n* Tree ensembles\n* Deprecate old tree code\n* Move DecisionTree implementation code to new API.\n* Move tests from the old suites which test the internals.\n* Update programming guide\n* Python API\n* Change RandomForest* to always use bootstrapping, even when numTrees = 1\n* Provide the probability of the predicted label for classification.  After we move code to the new API and update it to maintain probabilities for all labels, then we can add the probabilities to the new API.\n\nCC: mengxr  manishamde  codedeft  chouqin  MechCoder\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #5530 from jkbradley/dt-api-dt and squashes the following commits:\n\n6aae255 [Joseph K. Bradley] Changed tree abstractions not to take type parameters, and for setters to return this.type instead\nec17947 [Joseph K. Bradley] Updates based on code review.  Main changes were: moving public types from ml.impl.tree to ml.tree, modifying CategoricalSplit to take an Array of categories but store a Set internally, making more types sealed or final\n5626c81 [Joseph K. Bradley] style fixes\nf8fbd24 [Joseph K. Bradley] imported reorg of DecisionTreeSuite from old PR.  small cleanups\n7ef63ed [Joseph K. Bradley] Added DecisionTreeRegressor, test suites, and example (for real this time)\ne11673f [Joseph K. Bradley] Added DecisionTreeRegressor, test suites, and example\n119f407 [Joseph K. Bradley] added DecisionTreeClassifier example\n0bdc486 [Joseph K. Bradley] fixed issues after param PR was merged\nf9fbb60 [Joseph K. Bradley] Done with DecisionTreeClassifier, but no save/load yet.  Need to add example as well\n2532c9a [Joseph K. Bradley] partial move to spark.ml API, not done yet\nc72c1a0 [Joseph K. Bradley] Copied changes for common items, plus DecisionTreeClassifier from original PR","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/a83571acc938582865efb41645aa1e414f339e46"},{"sha":"59e206deb7346148412bbf5ba4ab626718fadf18","author":{"email":"be84b2e170174930486894ed46652314e5179dc8@alteryx.com","name":"cafreeman"},"message":"[SPARK-6807] [SparkR] Merge recent SparkR-pkg changes\n\nThis PR pulls in recent changes in SparkR-pkg, including\n\ncartesian, intersection, sampleByKey, subtract, subtractByKey, except, and some API for StructType and StructField.\n\nAuthor: cafreeman <cfreeman@alteryx.com>\nAuthor: Davies Liu <davies@databricks.com>\nAuthor: Zongheng Yang <zongheng.y@gmail.com>\nAuthor: Shivaram Venkataraman <shivaram.venkataraman@gmail.com>\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\nAuthor: Sun Rui <rui.sun@intel.com>\n\nCloses #5436 from davies/R3 and squashes the following commits:\n\nc2b09be [Davies Liu] SQLTypes -> schema\na5a02f2 [Davies Liu] Merge branch 'master' of github.com:apache/spark into R3\n168b7fe [Davies Liu] sort generics\nb1fe460 [Davies Liu] fix conflict in README.md\ne74c04e [Davies Liu] fix schema.R\n4f5ac09 [Davies Liu] Merge branch 'master' of github.com:apache/spark into R5\n41f8184 [Davies Liu] rm man\nae78312 [Davies Liu] Merge pull request #237 from sun-rui/SPARKR-154_3\n1bdcb63 [Zongheng Yang] Updates to README.md.\n5a553e7 [cafreeman] Use object attribute instead of argument\n71372d9 [cafreeman] Update docs and examples\n8526d2e71 [cafreeman] Remove `tojson` functions\n6ef5f2d [cafreeman] Fix spacing\n7741d66 [cafreeman] Rename the SQL DataType function\n141efd8 [Shivaram Venkataraman] Merge pull request #245 from hqzizania/upstream\n9387402 [Davies Liu] fix style\n40199eb [Shivaram Venkataraman] Move except into sorted position\n07d0dbc [Sun Rui] [SPARKR-244] Fix test failure after integration of subtract() and subtractByKey() for RDD.\n7e8caa3 [Shivaram Venkataraman] Merge pull request #246 from hlin09/fixCombineByKey\ned66c81 [cafreeman] Update `subtract` to work with `generics.R`\nf3ba785 [cafreeman] Fixed duplicate export\n275deb4 [cafreeman] Update `NAMESPACE` and tests\n1a3b63d [cafreeman] new version of `CreateDF`\n836c4bf [cafreeman] Update `createDataFrame` and `toDF`\nbe5d5c1 [cafreeman] refactor schema functions\n40338a4 [Zongheng Yang] Merge pull request #244 from sun-rui/SPARKR-154_5\n20b97a6 [Zongheng Yang] Merge pull request #234 from hqzizania/assist\nba54e34 [Shivaram Venkataraman] Merge pull request #238 from sun-rui/SPARKR-154_4\nc9497a3 [Shivaram Venkataraman] Merge pull request #208 from lythesia/master\nb317aa7 [Zongheng Yang] Merge pull request #243 from hqzizania/master\n136a07e [Zongheng Yang] Merge pull request #242 from hqzizania/stats\ncd66603 [cafreeman] new line at EOF\n8b76e81 [Shivaram Venkataraman] Merge pull request #233 from redbaron/fail-early-on-missing-dep\n7dd81b7 [cafreeman] Documentation\n0e2a94f [cafreeman] Define functions for schema and fields","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/59e206deb7346148412bbf5ba4ab626718fadf18"},{"sha":"d305e686b3d73213784bd75cdad7d168b22a1dc4","author":{"email":"c43522d98ec09bb312019761aae81416e0ca1a6b@lateral-thoughts.com","name":"Olivier Girardot"},"message":"SPARK-6988 : Fix documentation regarding DataFrames using the Java API\n\nThis patch includes :\n * adding how to use map after an sql query using javaRDD\n * fixing the first few java examples that were written in Scala\n\nThank you for your time,\n\nOlivier.\n\nAuthor: Olivier Girardot <o.girardot@lateral-thoughts.com>\n\nCloses #5564 from ogirardot/branch-1.3 and squashes the following commits:\n\n9f8d60e [Olivier Girardot] SPARK-6988 : Fix documentation regarding DataFrames using the Java API\n\n(cherry picked from commit 6b528dc139da594ef2e651d84bd91fe0f738a39d)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/d305e686b3d73213784bd75cdad7d168b22a1dc4"},{"sha":"a452c59210cf2c8ff8601cdb11401eea6dc14973","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Minor fix to SPARK-6958: Improve Python docstring for DataFrame.sort.\n\nAs a follow up PR to #5544.\n\ncc davies\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #5558 from rxin/sort-doc-improvement and squashes the following commits:\n\nf4c276f [Reynold Xin] Review feedback.\nd2dcf24 [Reynold Xin] Minor fix to SPARK-6958: Improve Python docstring for DataFrame.sort.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/a452c59210cf2c8ff8601cdb11401eea6dc14973"},{"sha":"c5ed510135aee3a1a0402057b3b5229892aa6f3a","author":{"email":"17d07ce14ed8a23e11d276b3691423f0ca203e51@capitalone.com","name":"Ilya Ganelin"},"message":"[SPARK-6703][Core] Provide a way to discover existing SparkContext's\n\nI've added a static getOrCreate method to the static SparkContext object that allows one to either retrieve a previously created SparkContext or to instantiate a new one with the provided config. The method accepts an optional SparkConf to make usage intuitive.\n\nStill working on a test for this, basically want to create a new context from scratch, then ensure that subsequent calls don't overwrite that.\n\nAuthor: Ilya Ganelin <ilya.ganelin@capitalone.com>\n\nCloses #5501 from ilganeli/SPARK-6703 and squashes the following commits:\n\ndb9a963 [Ilya Ganelin] Closing second spark context\n1dc0444 [Ilya Ganelin] Added ref equality check\n8c884fa [Ilya Ganelin] Made getOrCreate synchronized\ncb0c6b7 [Ilya Ganelin] Doc updates and code cleanup\n270cfe3 [Ilya Ganelin] [SPARK-6703] Documentation fixes\n15e8dea [Ilya Ganelin] Updated comments and added MiMa Exclude\n0e1567c [Ilya Ganelin] Got rid of unecessary option for AtomicReference\ndfec4da [Ilya Ganelin] Changed activeContext to AtomicReference\n733ec9f [Ilya Ganelin] Fixed some bugs in test code\n8be2f83 [Ilya Ganelin] Replaced match with if\ne92caf7 [Ilya Ganelin] [SPARK-6703] Added test to ensure that getOrCreate both allows creation, retrieval, and a second context if desired\na99032f [Ilya Ganelin] Spacing fix\nd7a06b8 [Ilya Ganelin] Updated SparkConf class to add getOrCreate method. Started test suite implementation","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/c5ed510135aee3a1a0402057b3b5229892aa6f3a"},{"sha":"6fbeb82e13db7117d8f216e6148632490a4bc5be","author":{"email":"580258f4dc0f7b7fba6752c3e796a6cb1bdd91fd@gmail.com","name":"Jongyoul Lee"},"message":"[SPARK-6350][Mesos] Make mesosExecutorCores configurable in mesos \"fine-grained\" mode\n\n- Defined executorCores from \"spark.mesos.executor.cores\"\n- Changed the amount of mesosExecutor's cores to executorCores.\n- Added new configuration option on running-on-mesos.md\n\nAuthor: Jongyoul Lee <jongyoul@gmail.com>\n\nCloses #5063 from jongyoul/SPARK-6350 and squashes the following commits:\n\n9238d6e [Jongyoul Lee] [SPARK-6350][Mesos] Make mesosExecutorCores configurable in mesos \"fine-grained\" mode - Fixed docs - Changed configuration name - Made mesosExecutorCores private\n2d41241 [Jongyoul Lee] [SPARK-6350][Mesos] Make mesosExecutorCores configurable in mesos \"fine-grained\" mode - Fixed docs\n89edb4f [Jongyoul Lee] [SPARK-6350][Mesos] Make mesosExecutorCores configurable in mesos \"fine-grained\" mode - Fixed docs\n8ba7694 [Jongyoul Lee] [SPARK-6350][Mesos] Make mesosExecutorCores configurable in mesos \"fine-grained\" mode - Fixed docs\n7549314 [Jongyoul Lee] [SPARK-6453][Mesos] Some Mesos*Suite have a different package with their classes - Fixed docs\n4ae7b0c [Jongyoul Lee] [SPARK-6453][Mesos] Some Mesos*Suite have a different package with their classes - Removed TODO\nc27efce [Jongyoul Lee] [SPARK-6453][Mesos] Some Mesos*Suite have a different package with their classes - Fixed Mesos*Suite for supporting integer WorkerOffers - Fixed Documentation\n1fe4c03 [Jongyoul Lee] [SPARK-6453][Mesos] Some Mesos*Suite have a different package with their classes - Change available resources of cpus to integer value beacuse WorkerOffer support the amount cpus as integer value\n5f3767e [Jongyoul Lee] Revert \"[SPARK-6350][Mesos] Make mesosExecutorCores configurable in mesos \"fine-grained\" mode\"\n4b7c69e [Jongyoul Lee] [SPARK-6350][Mesos] Make mesosExecutorCores configurable in mesos \"fine-grained\" mode - Changed configruation name and description from \"spark.mesos.executor.cores\" to \"spark.executor.frameworkCores\"\n0556792 [Jongyoul Lee] [SPARK-6350][Mesos] Make mesosExecutorCores configurable in mesos \"fine-grained\" mode - Defined executorCores from \"spark.mesos.executor.cores\" - Changed the amount of mesosExecutor's cores to executorCores. - Added new configuration option on running-on-mesos.md","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/6fbeb82e13db7117d8f216e6148632490a4bc5be"},{"sha":"1991337336596f94698e79c2366f065c374128ab","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-5933] [core] Move config deprecation warnings to SparkConf.\n\nI didn't find many deprecated configs after a grep-based search,\nbut the ones I could find were moved to the centralized location\nin SparkConf.\n\nWhile there, I deprecated a couple more HS configs that mentioned\ntime units.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #5562 from vanzin/SPARK-5933 and squashes the following commits:\n\ndcb617e7 [Marcelo Vanzin] [SPARK-5933] [core] Move config deprecation warnings to SparkConf.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/1991337336596f94698e79c2366f065c374128ab"}]}