How do I resolve the error that I receive in CloudFormation when I try to publish slow logs to CloudWatch Logs?
I want to resolve the error that I receive in AWS CloudFormation when I try to publish slow logs to Amazon CloudWatch Logs. The error is: "The Resource Access Policy specified for the CloudWatch Logs log group /aws/aes/domains/search/search-logs does not grant sufficient permissions for Amazon Elasticsearch Service to create a log stream."
"I want to resolve the error that I receive in AWS CloudFormation when I try to publish slow logs to Amazon CloudWatch Logs. The error is: "The Resource Access Policy specified for the CloudWatch Logs log group /aws/aes/domains/search/search-logs does not grant sufficient permissions for Amazon Elasticsearch Service to create a log stream."Short descriptionTo resolve this error, use a separate policy at the log group level to allow Amazon Elasticsearch Service (Amazon ES) to push logs to CloudWatch Logs. Then, use AccessPolicies in the AWS::Elasticsearch::Domain resource to set permissions for Amazon ES domains.The following steps show you how to publish slow logs to CloudWatch with CloudFormation by using an AWS Lambda-backed custom resource created in Python 3.6. The custom resource triggers a Lambda function, which triggers the PutResourcePolicy API to publish slow logs.Note: When CloudFormation enables log publishing to CloudWatch, the AWS::Logs::LogGroup resource doesn't have a property to assign a resource access policy. The resource access policy specified for the CloudWatch Logs log group should grant sufficient permissions for Amazon ES to publish the log stream. You can't create an access policy permission directly using a CloudFormation resource. This is because the PutResourcePolicy API call for the AWS::Logs::LogGroup resource isn't supported by CloudFormation.ResolutionNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.The following CloudFormation template uses a custom resource to get the name of the log group. Then, the template applies the policy that allows the Amazon ES service to make API calls on the log group.1.    Create a CloudFormation template called ESlogsPermission.yaml:Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.Permission is hereby granted, free of charge, to any person obtaining a copy of thissoftware and associated documentation files (the "Software"), to deal in the Softwarewithout restriction, including without limitation the rights to use, copy, modify,merge, publish, distribute, sublicense, and/or sell copies of the Software, and topermit persons to whom the Software is furnished to do so.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR APARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHTHOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTIONOF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THESOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.AWSTemplateFormatVersion: 2010-09-09Description: AWS cloudFormation template to publish slow logs to Amazon CloudWatch Logs.Parameters:  LogGroupName:    Type: String    Description: Please don't change the log group name while updating  ESDomainName:    Description: A name for the Amazon Elastic Search domain    Type: String  LambdaFunctionName:    Description: Lambda Function Name    Type: StringResources:  AwsLogGroup:    Type: 'AWS::Logs::LogGroup'    Properties:      LogGroupName: !Ref LogGroupName  LambdaLogGroup:    Type: 'AWS::Logs::LogGroup'    Properties:      LogGroupName: !Sub '/aws/lambda/${LambdaFunctionName}'  LambdaExecutionRole:    Type: 'AWS::IAM::Role'    Properties:      AssumeRolePolicyDocument:        Version: 2012-10-17        Statement:          - Effect: Allow            Principal:              Service:                - lambda.amazonaws.com            Action:              - 'sts:AssumeRole'      Path: /      Policies:        - PolicyName: root1          PolicyDocument:            Version: 2012-10-17            Statement:              - Effect: Allow                Action:                  - 'logs:CreateLogStream'                  - 'logs:PutLogEvents'                Resource: !Sub >-                  arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/${LambdaFunctionName}:log-stream:*        - PolicyName: root2          PolicyDocument:            Version: 2012-10-17            Statement:              - Effect: Allow                Action:                  - 'logs:CreateLogGroup'                Resource:                  - !Sub >-                    arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/${LambdaFunctionName}                  - !Sub >-                    arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/${LogGroupName}              - Effect: Allow                Action:                  - 'logs:PutResourcePolicy'                  - 'logs:DeleteResourcePolicy'                Resource: !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:*'  logGroupPolicyFunction:    DependsOn: LambdaLogGroup    Type: 'AWS::Lambda::Function'    Properties:      FunctionName: !Ref LambdaFunctionName      Code:        ZipFile: >          import urllib3          import json          import boto3          http = urllib3.PoolManager()          SUCCESS = "SUCCESS"          FAILED = "FAILED"          def send(event, context, responseStatus, responseData,          physicalResourceId=None, noEcho=False):              responseUrl = event['ResponseURL']              print(responseUrl)              responseBody = {}              responseBody['Status'] = responseStatus              responseBody['Reason'] = 'See the details in CloudWatch Log Stream: ' + context.log_stream_name              responseBody['PhysicalResourceId'] = physicalResourceId or context.log_stream_name              responseBody['StackId'] = event['StackId']              responseBody['RequestId'] = event['RequestId']              responseBody['LogicalResourceId'] = event['LogicalResourceId']              responseBody['NoEcho'] = noEcho              responseBody['Data'] = responseData              json_responseBody = json.dumps(responseBody)              print("Response body:\n" + json_responseBody)              headers = {                  'content-type' : '',                  'content-length' : str(len(json_responseBody))              }              try:                  response = http.request('PUT',responseUrl,body=json_responseBody.encode('utf-8'),headers=headers)                  print("Status code: " + response.reason)              except Exception as e:                  print("send(..) failed executing requests.put(..): " + str(e))          def handler(event, context):              logsgroup_policy_name=event['ResourceProperties']['CWLOGS_NAME']              cw_log_group_arn=event['ResourceProperties']['CWLOG_ARN']              cwlogs = boto3.client('logs')              loggroup_policy={                  "Version": "2012-10-17",                  "Statement": [{                  "Sid": "",                  "Effect": "Allow",                  "Principal": { "Service": "es.amazonaws.com"},                  "Action":[                  "logs:PutLogEvents",                  " logs:PutLogEventsBatch",                  "logs:CreateLogStream"                      ],                  'Resource': f'{cw_log_group_arn}'                  }]                  }              loggroup_policy = json.dumps(loggroup_policy)              if(event['RequestType'] == 'Delete'):                  print("Request Type:",event['RequestType'])                  cwlogs.delete_resource_policy(                  policyName=logsgroup_policy_name                 )                  responseData={}                  send(event, context, SUCCESS, responseData)              elif(event['RequestType'] == 'Create'):                  try:                      cwlogs.put_resource_policy(                      policyName = logsgroup_policy_name,                      policyDocument = loggroup_policy                       )                      responseData={}                      print("Sending response to custom resource")                      send(event, context, SUCCESS, responseData)                  except Exception as  e:                      print('Failed to process:', e)                      send(event, context, FAILED, responseData)              elif(event['RequestType'] == 'Update'):                  try:                      responseData={}                      print("Update is not supported on this resource")                      send(event, context, SUCCESS, responseData)                  except Exception as  e:                      print('Failed to process:', e)                      send(event, context, FAILED, responseData)      Handler: index.handler      Role: !GetAtt         - LambdaExecutionRole        - Arn      Runtime: python3.6  logGroupPolicycustomresource:    Type: 'Custom::LogGroupPolicy'    Properties:      ServiceToken: !GetAtt         - logGroupPolicyFunction        - Arn      CWLOGS_NAME: !Ref LogGroupName      CWLOG_ARN: !GetAtt         - AwsLogGroup        - Arn  ElasticsearchDomain:    Type: 'AWS::Elasticsearch::Domain'    DependsOn: logGroupPolicycustomresource    Properties:      DomainName: !Ref ESDomainName      ElasticsearchVersion: '6.2'      EBSOptions:        EBSEnabled: true        VolumeSize: 10        VolumeType: gp2      LogPublishingOptions:        SEARCH_SLOW_LOGS:          CloudWatchLogsLogGroupArn: !GetAtt             - AwsLogGroup            - Arn          Enabled: true2.    To launch a CloudFormation stack with the ESlogsPermission.yaml file, use the CloudFormation console or the following AWS CLI command:aws cloudformation create-stack --stack-name yourStackName --template-body file://yourTemplateName --parameters ParameterKey=LogGroupName,ParameterValue=Your-LogGroup-Name, ParameterKey=ESDomainName,ParameterValue=Your-ES-Name --capabilities CAPABILITY_NAMED_IAM --region yourRegionNote: Replace yourStackName, yourTemplateName, Your-LogGroup-Name, Your-ES-Name, and yourRegion with your values.The CloudFormation template does the following for you:1.    Creates a log group.2.    Creates a Lambda function. The Lambda function gets the log group name from the Parameters section of the CloudFormation template by using a custom resource. The Lambda function calls the PutResourcePolicy API for the log group name. The log group must have a policy to allow the Amazon ES domain to put the logs.3.    Creates a Lambda-backed custom resource to invoke the Lambda function created in step 2. The custom resource helps apply PutResourcePolicy on the log group Amazon Resource Name (ARN) so that Amazon ES can stream logs. In the template, CloudFormation uses a custom resource to create an Amazon ES domain with the LogPublishingOption.Follow"