Why does my AWS Glue job fail with the error "Temporary directory not specified" when I insert or extract data from Amazon Redshift?
My AWS Glue job fails with the error "Temporary directory not specified" when I insert or extract data from Amazon Redshift.
"My AWS Glue job fails with the error "Temporary directory not specified" when I insert or extract data from Amazon Redshift.Short descriptionHere are a few things to remember when your AWS Glue job writes or reads data from Amazon Redshift:Your AWS Glue job writes data into an Amazon Redshift cluster: The job initially writes the data into an Amazon Simple Storage Service (Amazon S3) bucket in CSV format. Then, the job issues a COPY command to Amazon Redshift.Your AWS Glue job reads data from an Amazon Redshift cluster: The job first unloads the data into an Amazon S3 bucket in CSV format using the UNLOAD command. Then, the job loads the data into the DynamicFrame from these temporary bucket files.You might get this error either when either of the following conditions are true:You are unloading the data from Amazon Redshift into the temporary S3 bucket.You are loading the data from the S3 bucket to Amazon Redshift using the COPY or UNLOAD command.ResolutionThe following are some of the common causes and solution options for this error.Define a temporary directoryThe most common reason for this error is the missing temporary S3 bucket that's used by the AWS Glue job as a staging directory. Therefore, be sure to define an S3 bucket as the temporary directory for your job. For more information on how to define a temporary bucket, see Special parameters used by AWS Glue.Verify the IAM role permissionsVerify the IAM role permissions to be sure that you have the right permissions to access the temporary S3 bucket. Also, be sure that you didn't block required permissions for the bucket in the following policies for the AWS Glue IAM role:Bucket policyS3 VPC endpoint policyAWS Organizations policyService control policySome examples of required permissions are ListObjects, GetObject, and PutObject.Verify the name of the temporary directoryBe sure that the name of the S3 bucket that's used as the temporary directory doesn't have a period in it to avoid getting the following exception:Caused by: java.sql.SQLException: [Amazon](500310) Invalid operation: UNLOAD destination is not supported.Verify AWS Key Management Service (AWS KMS) permissionsIf you use customer managed keys from AWS Key Management Service (AWS KMS) to encrypt your data, then be sure to do the include extraunloadoptions in additional_options for your ETL statement in the AWS Glue script. For example:datasource0 = glueContext.create_dynamic_frame.from_catalog(    database = "database-name",     table_name = "table-name",     redshift_tmp_dir = args["TempDir"],    additional_options = {"extraunloadoptions":"ENCRYPTED KMS_KEY_ID 'CMK key ID'"},     transformation_ctx = "datasource0"  )If you are using AWS KMS to encrypt S3 data and facing permission issues related to AWS KMS, then be sure of the following:You have the AWS KMS action permissions similar to the following in the AWS Glue IAM role.You added the AWS Glue IAM role into the AWS KMS key."Action": [    "kms:Encrypt",    "kms:Decrypt",    "kms:ReEncrypt*",    "kms:GenerateDataKey*",    "kms:DescribeKey"    ]Verify the IAM role permissions for AWS Glue Python Shell jobIf you are trying to run the COPY or UNLOAD command from an AWS Glue Python Shell job and can't load the credentials, then be sure of the following:You added the AWS Glue IAM role in Amazon Redshift.The AWS Glue job role includes Amazon Redshift and AWS Glue in its trust relationship policy. For example:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "Service": [          "glue.amazonaws.com",          "redshift.amazonaws.com"        ]      },      "Action": "sts:AssumeRole"    }  ]}Check for packet dropThe failure of your queries to reach the Amazon Redshift cluster even after the connection is successful might be due to the maximum transmission unit (MTU) size mismatch between Amazon Redshift and AWS Glue network path. Try configuring Amazon Redshift security groups to allow ICMP "destination unreachable". For more information, see Queries appear to hang and sometimes fail to reach the cluster.Define the temporary directory in the AWS CloudFormation templateIf you created your AWS Glue job using CloudFormation, then be sure that you provided the temporary directory location in the DefaultArguments parameter in your CloudFormation template. For example:"DefaultArguments": { "--TempDir": "s3://doc-example-bucket/doc-example-folder"}Define temporary directory in the DynamicFrameIf you get the error even after defining the temporary directory in your AWS Glue job, then check if you are reading or writing into Amazon Redshift. You can do so using the AWS Glue's DynamicFrame method. Then, confirm the following:You passed your temporary directory into the redshift_tmp_dir property of your DynamicFrame:To create a DynamicFrame using a Data Catalog database and table, see create_dynamic_frame_from_catalog.To create a DynamicFrame with a specified connection and format, see create_dynamic_frame_from_options.To write into a DynamicFrame using information from a Data Catalog database and table, see write_dynamic_frame_from_catalog.To write into a DynamicFrame with a specified connection and format, see write_dynamic_frame_from_options.To write into a DynamicFrame or DynamicFrameCollection with a specified connection and format see write_from_options.To write into a DynamicFrame or DynamicFrameCollection using information from a specified JDBC connection, see write_from_jdbc_conf.You have TempDir specified in the getResolvedOptions function for your ETL job:Use the following commands to retrieve the job name and TempDir parameters:import sysfrom awsglue.utils import getResolvedOptionsargs = getResolvedOptions(sys.argv,['JOB_NAME','TempDir'])TempDir = args['TempDir']For more information, see Accessing parameters using getResolvedOptions.Related informationMoving data to and from Amazon RedshiftFollow"