{"push_id":778332173,"size":4,"distinct_size":4,"ref":"refs/heads/master","head":"00d9af5e190475affffb8b50467fcddfc40f50dc","before":"e6e483cc4de740c46398385b03ffe0e662edae39","commits":[{"sha":"3f63bd6023edcc9af268933a235f34e10bc3d2ba","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-10398] [DOCS] Migrate Spark download page to use new lua mirroring scripts\n\nMigrate Apache download closer.cgi refs to new closer.lua\n\nThis is the bit of the change that affects the project docs; I'm implementing the changes to the Apache site separately.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #8557 from srowen/SPARK-10398.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/3f63bd6023edcc9af268933a235f34e10bc3d2ba"},{"sha":"ec012805337926e56343be2761a1037296446880","author":{"email":"02e10e341a2fea4868ba58fdfcfe400dcd4d1dd0@yahoo-inc.com","name":"zhuol"},"message":"[SPARK-4223] [CORE] Support * in acls.\n\nSPARK-4223.\n\nCurrently we support setting view and modify acls but you have to specify a list of users. It would be nice to support * meaning all users have access.\n\nManual tests to verify that: \"*\" works for any user in:\na. Spark ui: view and kill stage.     Done.\nb. Spark history server.                  Done.\nc. Yarn application killing.  Done.\n\nAuthor: zhuol <zhuol@yahoo-inc.com>\n\nCloses #8398 from zhuoliu/4223.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/ec012805337926e56343be2761a1037296446880"},{"sha":"bf550a4b551b6dd18fea3eb3f70497f9a6ad8e6c","author":{"email":"ea6ccdfbfefd2f7cfa3f93f15ce5c28ccba7db78@gmail.com","name":"0x0FFF"},"message":"[SPARK-10162] [SQL] Fix the timezone omitting for PySpark Dataframe filter function\n\nThis PR addresses [SPARK-10162](https://issues.apache.org/jira/browse/SPARK-10162)\nThe issue is with DataFrame filter() function, if datetime.datetime is passed to it:\n* Timezone information of this datetime is ignored\n* This datetime is assumed to be in local timezone, which depends on the OS timezone setting\n\nFix includes both code change and regression test. Problem reproduction code on master:\n```python\nimport pytz\nfrom datetime import datetime\nfrom pyspark.sql import *\nfrom pyspark.sql.types import *\nsqc = SQLContext(sc)\ndf = sqc.createDataFrame([], StructType([StructField(\"dt\", TimestampType())]))\n\nm1 = pytz.timezone('UTC')\nm2 = pytz.timezone('Etc/GMT+3')\n\ndf.filter(df.dt > datetime(2000, 01, 01, tzinfo=m1)).explain()\ndf.filter(df.dt > datetime(2000, 01, 01, tzinfo=m2)).explain()\n```\nIt gives the same timestamp ignoring time zone:\n```\n>>> df.filter(df.dt > datetime(2000, 01, 01, tzinfo=m1)).explain()\nFilter (dt#0 > 946713600000000)\n Scan PhysicalRDD[dt#0]\n\n>>> df.filter(df.dt > datetime(2000, 01, 01, tzinfo=m2)).explain()\nFilter (dt#0 > 946713600000000)\n Scan PhysicalRDD[dt#0]\n```\nAfter the fix:\n```\n>>> df.filter(df.dt > datetime(2000, 01, 01, tzinfo=m1)).explain()\nFilter (dt#0 > 946684800000000)\n Scan PhysicalRDD[dt#0]\n\n>>> df.filter(df.dt > datetime(2000, 01, 01, tzinfo=m2)).explain()\nFilter (dt#0 > 946695600000000)\n Scan PhysicalRDD[dt#0]\n```\nPR [8536](https://github.com/apache/spark/pull/8536) was occasionally closed by me dropping the repo\n\nAuthor: 0x0FFF <programmerag@gmail.com>\n\nCloses #8555 from 0x0FFF/SPARK-10162.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/bf550a4b551b6dd18fea3eb3f70497f9a6ad8e6c"},{"sha":"00d9af5e190475affffb8b50467fcddfc40f50dc","author":{"email":"ea6ccdfbfefd2f7cfa3f93f15ce5c28ccba7db78@gmail.com","name":"0x0FFF"},"message":"[SPARK-10392] [SQL] Pyspark - Wrong DateType support on JDBC connection\n\nThis PR addresses issue [SPARK-10392](https://issues.apache.org/jira/browse/SPARK-10392)\nThe problem is that for \"start of epoch\" date (01 Jan 1970) PySpark class DateType returns 0 instead of the `datetime.date` due to implementation of its return statement\n\nIssue reproduction on master:\n```\n>>> from pyspark.sql.types import *\n>>> a = DateType()\n>>> a.fromInternal(0)\n0\n>>> a.fromInternal(1)\ndatetime.date(1970, 1, 2)\n```\n\nAuthor: 0x0FFF <programmerag@gmail.com>\n\nCloses #8556 from 0x0FFF/SPARK-10392.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/00d9af5e190475affffb8b50467fcddfc40f50dc"}]}