{"push_id":739469962,"size":23,"distinct_size":23,"ref":"refs/heads/unsafe-by-default","head":"8946cb96cbe1c435198891432648fc4c262a583a","before":"a2ccefa70d542f947c6cd6cc5815286d4f730f05","commits":[{"sha":"ce89ff477aea6def68265ed218f6105680755c9a","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-9386] [SQL] Feature flag for metastore partition pruning\n\nSince we have been seeing a lot of failures related to this new feature, lets put it behind a flag and turn it off by default.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #7703 from marmbrus/optionalMetastorePruning and squashes the following commits:\n\n6ad128c [Michael Armbrust] style\n8447835 [Michael Armbrust] [SPARK-9386][SQL] Feature flag for metastore partition pruning\nfd37b87 [Michael Armbrust] add config flag","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/ce89ff477aea6def68265ed218f6105680755c9a"},{"sha":"daa1964b6098f79100def78451bda181b5c92198","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-8882] [STREAMING] Add a new Receiver scheduling mechanism\n\nThe design doc: https://docs.google.com/document/d/1ZsoRvHjpISPrDmSjsGzuSu8UjwgbtmoCTzmhgTurHJw/edit?usp=sharing\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #7276 from zsxwing/receiver-scheduling and squashes the following commits:\n\n137b257 [zsxwing] Add preferredNumExecutors to rescheduleReceiver\n61a6c3f [zsxwing] Set state to ReceiverState.INACTIVE in deregisterReceiver\n5e1fa48 [zsxwing] Fix the code style\n7451498 [zsxwing] Move DummyReceiver back to ReceiverTrackerSuite\n715ef9c [zsxwing] Rename: scheduledLocations -> scheduledExecutors; locations -> executors\n05daf9c [zsxwing] Use receiverTrackingInfo.toReceiverInfo\n1d6d7c8 [zsxwing] Merge branch 'master' into receiver-scheduling\n8f93c8d [zsxwing] Use hostPort as the receiver location rather than host; fix comments and unit tests\n59f8887 [zsxwing] Schedule all receivers at the same time when launching them\n075e0a3 [zsxwing] Add receiver RDD name; use '!isTrackerStarted' instead\n276a4ac [zsxwing] Remove \"ReceiverLauncher\" and move codes to \"launchReceivers\"\nfab9a01 [zsxwing] Move methods back to the outer class\n4e639c4 [zsxwing] Fix unintentional changes\nf60d021 [zsxwing] Reorganize ReceiverTracker to use an event loop for lock free\n105037e [zsxwing] Merge branch 'master' into receiver-scheduling\n5fee132 [zsxwing] Update tha scheduling algorithm to avoid to keep restarting Receiver\n9e242c8 [zsxwing] Remove the ScheduleReceiver message because we can refuse it when receiving RegisterReceiver\na9acfbf [zsxwing] Merge branch 'squash-pr-6294' into receiver-scheduling\n881edb9 [zsxwing] ReceiverScheduler -> ReceiverSchedulingPolicy\ne530bcc [zsxwing] [SPARK-5681][Streaming] Use a lock to eliminate the race condition when stopping receivers and registering receivers happen at the same time #6294\n3b87e4a [zsxwing] Revert SparkContext.scala\na86850c [zsxwing] Remove submitAsyncJob and revert JobWaiter\nf549595 [zsxwing] Add comments for the scheduling approach\n9ecc08e [zsxwing] Fix comments and code style\n28d1bee [zsxwing] Make 'host' protected; rescheduleReceiver -> getAllowedLocations\n2c86a9e [zsxwing] Use tryFailure to support calling jobFailed multiple times\nca6fe35 [zsxwing] Add a test for Receiver.restart\n27acd45 [zsxwing] Add unit tests for LoadBalanceReceiverSchedulerImplSuite\ncc76142 [zsxwing] Add JobWaiter.toFuture to avoid blocking threads\nd9a3e72 [zsxwing] Add a new Receiver scheduling mechanism","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/daa1964b6098f79100def78451bda181b5c92198"},{"sha":"2e7f99a004f08a42e86f6f603e4ba35cb52561c4","author":{"email":"c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","name":"Daoyuan Wang"},"message":"[SPARK-8195] [SPARK-8196] [SQL] udf next_day last_day\n\nnext_day, returns next certain dayofweek.\nlast_day, returns the last day of the month which given date belongs to.\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #6986 from adrian-wang/udfnlday and squashes the following commits:\n\nef7e3da [Daoyuan Wang] fix\n02b3426 [Daoyuan Wang] address 2 comments\ndc69630 [Daoyuan Wang] address comments from rxin\n8846086 [Daoyuan Wang] address comments from rxin\nd09bcce [Daoyuan Wang] multi fix\n1a9de3d [Daoyuan Wang] function next_day and last_day","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/2e7f99a004f08a42e86f6f603e4ba35cb52561c4"},{"sha":"84da8792e2a99736edb6c94df7eda87915a8a476","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9395][SQL] Create a SpecializedGetters interface to track all the specialized getters.\n\nAs we are adding more and more specialized getters to more classes (coming soon ArrayData), this interface can help us prevent missing a method in some interfaces.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #7713 from rxin/SpecializedGetters and squashes the following commits:\n\n3b39be1 [Reynold Xin] Added override modifier.\n567ba9c [Reynold Xin] [SPARK-9395][SQL] Create a SpecializedGetters interface to track all the specialized getters.","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/84da8792e2a99736edb6c94df7eda87915a8a476"},{"sha":"3bc7055e265ee5c75af8726579663cea0590f6c0","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Fixed a test failure.","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/3bc7055e265ee5c75af8726579663cea0590f6c0"},{"sha":"63a492b931765b1edd66624421d503f1927825ec","author":{"email":"348ff379544b45aa3f16e53a543620141423a0f6@gmail.com","name":"Yijie Shen"},"message":"[SPARK-8828] [SQL] Revert SPARK-5680\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-8828\n\nAuthor: Yijie Shen <henry.yijieshen@gmail.com>\n\nCloses #7667 from yjshen/revert_combinesum_2 and squashes the following commits:\n\nc37ccb1 [Yijie Shen] add test case\n8377214 [Yijie Shen] revert spark.sql.useAggregate2 to its default value\ne2305ac [Yijie Shen] fix bug - avg on decimal column\n7cb0e95 [Yijie Shen] [wip] resolving bugs\n1fadb5a [Yijie Shen] remove occurance\n17c6248 [Yijie Shen] revert SPARK-5680","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/63a492b931765b1edd66624421d503f1927825ec"},{"sha":"60f08c7c8775c0462b74bc65b41397be6eb24b6d","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-9373][SQL] Support StructType in Tungsten projection\n\nThis pull request updates GenerateUnsafeProjection to support StructType. If an input struct type is backed already by an UnsafeRow, GenerateUnsafeProjection copies the bytes directly into its buffer space without any conversion. However, if the input is not an UnsafeRow, GenerateUnsafeProjection runs the code generated recursively to convert the input into an UnsafeRow and then copies it into the buffer space.\n\nAlso create a TungstenProject operator that projects data directly into UnsafeRow. Note that I'm not sure if this is the way we want to structure Unsafe+codegen operators, but we can defer that decision to follow-up pull requests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #7689 from rxin/tungsten-struct-type and squashes the following commits:\n\n9162f42 [Reynold Xin] Support IntervalType in UnsafeRow's getter.\nbe9f377 [Reynold Xin] Fixed tests.\n10c4b7c [Reynold Xin] Format generated code.\n77e8d0e [Reynold Xin] Fixed NondeterministicSuite.\nac4951d [Reynold Xin] Yay.\nac203bf [Reynold Xin] More comments.\n9f36216 [Reynold Xin] Updated comment.\n6b781fe [Reynold Xin] Reset the change in DataFrameSuite.\n525b95b [Reynold Xin] Merged with master, more documentation & test cases.\n321859a [Reynold Xin] [SPARK-9373][SQL] Support StructType in Tungsten projection [WIP]","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/60f08c7c8775c0462b74bc65b41397be6eb24b6d"},{"sha":"9c5612f4e197dec82a5eac9542896d6216a866b7","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[MINOR] [SQL] Support mutable expression unit test with codegen projection\n\nThis is actually contains 3 minor issues:\n1) Enable the unit test(codegen) for mutable expressions (FormatNumber, Regexp_Replace/Regexp_Extract)\n2) Use the `PlatformDependent.copyMemory` instead of the `System.arrayCopy`\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #7566 from chenghao-intel/codegen_ut and squashes the following commits:\n\n24f43ea [Cheng Hao] enable codegen for mutable expression & UTF8String performance","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/9c5612f4e197dec82a5eac9542896d6216a866b7"},{"sha":"713f7e5c1b24d73cea1624b31c3d35df54d2d9bb","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"Enable Unsafe by default","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/713f7e5c1b24d73cea1624b31c3d35df54d2d9bb"},{"sha":"06ee79469fea5d2342e4fe9c6d637e087c95a3f8","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"Do not use UnsafeExternalSort operator if codegen is disabled","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/06ee79469fea5d2342e4fe9c6d637e087c95a3f8"},{"sha":"1d33d5168382d9b68fc09c22ce72ca672b430ad0","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"Disable unsafe Exchange path when RangePartitioning is used","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/1d33d5168382d9b68fc09c22ce72ca672b430ad0"},{"sha":"1a74a5dec4adbbf91bdf87088bfe3f6617603e3c","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"Reduce page size to make HiveCompatibilitySuite pass.","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/1a74a5dec4adbbf91bdf87088bfe3f6617603e3c"},{"sha":"f8618310a5b45c847edc9bcd2d8e51c38c9c6cbc","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"Add task completion callback to avoid leak in limit after sort","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/f8618310a5b45c847edc9bcd2d8e51c38c9c6cbc"},{"sha":"8f82c53b79bae5cd887df56723cdea6765c291eb","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"Add basic unit test for script transform with 'cat' command.","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/8f82c53b79bae5cd887df56723cdea6765c291eb"},{"sha":"4200154062661851bc04cb4847d5becfaf678357","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"Clean up nullability in ScriptTransformation","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/4200154062661851bc04cb4847d5becfaf678357"},{"sha":"3517e0b566fd5dc5651ca7b0be8bb0d6a2349fed","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"Skip launching of external command for empty partitions.","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/3517e0b566fd5dc5651ca7b0be8bb0d6a2349fed"},{"sha":"aae64fa7c4dea9def00db4a15544f3d94f9b1cae","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"Kill script transform subprocess when error occurs in input writer.","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/aae64fa7c4dea9def00db4a15544f3d94f9b1cae"},{"sha":"b164bf31803891c384a8a9f37e3c5fe5438350de","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"Add failing test which demonstrates exception masking issue","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/b164bf31803891c384a8a9f37e3c5fe5438350de"},{"sha":"86c37bbcc91be75c8e15fca7fa2031239c55fa28","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"Split ScriptTransformation writer thread into own class.","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/86c37bbcc91be75c8e15fca7fa2031239c55fa28"},{"sha":"34f8580bef27dcc1a0f3ed4b345ae53b5196e303","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"Rename iterator variables to disambiguate.","distinct":true,"url":"https://api.github.com/repos/JoshRosen/spark/commits/34f8580bef27dcc1a0f3ed4b345ae53b5196e303"}]}