How do I connect to Amazon OpenSearch Service using Filebeat and Logstash on Amazon Linux?
"I'm trying to connect to an Amazon OpenSearch Service cluster using Logstash on Amazon Linux. However, I keep getting an error."
"I'm trying to connect to an Amazon OpenSearch Service cluster using Logstash on Amazon Linux. However, I keep getting an error.Short descriptionTo connect to Amazon OpenSearch Service using Logstash, perform the following steps:1.    Set up your security ports (such as port 443) to forward logs to OpenSearch Service.2.    Update your Filebeat, Logstash, and OpenSearch Service configurations.3.    Install Filebeat on your source Amazon Elastic Compute Cloud (Amazon EC2) instance. Make sure that you correctly install and configure your YAML config file.4.    Install Logstash on a separate Amazon EC2 instance where the logs will be sent from.If you didn't correctly set up or configured Logstash, then you receive one of these errors: 401 Authorization error, 403 Forbidden error, or x-pack installation error.ResolutionSet up your security portsMake sure to set up your security ports so that your EC2 instance can forward logs to OpenSearch Service.To set up your security ports to forward logs from Logstash, perform the following steps:1.    Create an EC2 instance where you installed Apache and Filebeat. The EC2 instance must be able to forward logs from Logstash to OpenSearch Service.2.    Make sure that your EC2 instances reside in the same security group as your virtual private cloud (VPC) for OpenSearch Service.3.    Make sure that the following ports are open in your security group: 80, 443, and 5044. These ports must be open so that you can send data between Logstash and OpenSearch Service.Update your Filebeat, Logstash, and OpenSearch Service configurationsOpenSearch Service runs best when you use the same OSS versions. Therefore, try to use compatible versions for the following:Filebeat version x.x OSSLogstash version x.x OSS (v7.16.2 & v6.8.22 for Log4j security patch)OpenSearch Service version x.xIn the preceding example, the "x.x" of the version numbers should be matching in your configurations.To make sure that the downloaded software remains in sync, download RPMs to each separate EC2 instance. To prevent a single point of failure in your pipeline, it's a best practice to avoid running Filebeat and Logstash service on the same EC2 instance.Note: If you're using OpenSearch Service versions 1.0 or higher, then make sure that compatibility mode is turned on when you first launch your domain. Note that you can turn on compatibility mode only through the AWS Management Console.Install Filebeat on the source EC2 instance1.    Download the RPM for the desired version of Filebeat:wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-oss-7.16.2-x86_64.rpm2.    Install the Filebeat RPM file:rpm -ivh filebeat-oss-7.16.2-x86_64.rpmInstall Logstash on a separate EC2 instance from which the logs will be sent1.    Download the RPM file of the desired Logstash version:wget https://artifacts.elastic.co/downloads/logstash/logstash-oss-7.16.2-x86_64.rpmThis example uses version 6.7 to match the version number of OpenSearch Service and Filebeat.2.    Install Java or OpenJDK on your EC2 instance before installing Logstash RPM file:yum install java-1.8.0-*3.    After installing Java, install the RPM file that you downloaded for Logstash using the rpm command:rpm -ivh logstash-oss-7.16.2-x86_64.rpmNote: Logstash requires Java to run. This example uses Java version 8 (Open JDK 1.8), which is supported by all versions of Logstash. For more information about the supported versions of Java and Logstash, see the Support matrix on the Elasticsearch website.4.    Verify the configuration files by checking the "/etc/filebeat" and "/etc/logstash" directories.5.    For Filebeat, update the output to either Logstash or OpenSearch Service, and specify that logs must be sent. Then, start your service.Note: If you try to upload templates to OpenSearch Dashboards with Filebeat, your upload fails. Filebeat assumes that your cluster has x-pack plugin support.6.    Update your Filebeat YAML configuration file to send Apache access logs to Logstash.For example:filebeat.inputs:- type: log  enabled: true  paths:    - /var/log/*.log filebeat.config.modules:  path: ${path.config}/modules.d/*.yml  reload.enabled: false setup.template.settings: index.number_of_shards: 1index.codec: best_compression #output.elasticsearch:#hosts: ["vpc-examplestack-5crrfyysa2ratcl3ursmung33q.us-east-1.es.amazonaws.com:443"]#protocol: "https" output.logstash:  # The Logstash hosts  hosts: ["Logstash-EC2-InstanceIP:5044"] setup.ilm.enabled: false ilm.enabled: false7.    Make sure that your Logstash configuration file can access Filebeat on Port 5044. This port access allows Logstash to forward requests to your OpenSearch Service VPC endpoint.For example:input {  beats {    port => 5044  }}output {  opensearch {    hosts => ["https://domain-endpoint:443"]    ssl => true    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"    user => "my-username"    password => "my-password"  }}Note: Logstash OSS version 7.12.1 supports both logstash-output-elasticsearch and logstash-output-opensearch plugins. Logstash OSS version 7.13 and after support logstash-output-opensearch plugin only.8.    Start the Filebeat and Logstash services with the following commands on each instance.Filebeat:systemctl start filebeat (service filebeat start)Logstash:cp /etc/logstash/logstash.conf /etc/logstash/conf.d/systemctl start logstash (service logstash start)9.    Run a cat indices API call to your OpenSearch Service domain to confirm that the Filebeat logs are being sent. If your logs are successfully sent, you receive the following response:curl -XGET https://vpc-examplestack-5crrfyysa2ratcl3ursmung33q.us-east-1.es.amazonaws.com/_cat/indicesgreen open filebeat-7.16.2-2022.01.27 f97c4WnuQ-CtsAJJaJHUlg1 1 1511515 0 249.7mb 124.7mbgreen open .kibana_1                 Ioco6fUoSCGkaOvHNCL39g 11       1 0   7.4kb  3.7kbBy default, the Filebeat indices rotate daily. Here's an example output of a Filebeat index:curl -XGET https://vpc-examplestack-5crrfyysa2ratcl3ursmung33q.us-east-1.es.amazonaws.com/_cat/indicesgreen open filebeat-7.16.2-2022.01.27 f97c4WnuQ-CtsAJJaJHUlg1 1 1511515 0 249.7mb 124.7mbgreen open .kibana_1                 Ioco6fUoSCGkaOvHNCL39g 11       1 0   7.4kb  3.7kbgreen open filebeat-7.16.2-2022.01.28 4i8W0smlRGGFcQOaDMxonA1 1      89 0 207.1kb 118.1kbIf you successfully configure Filebeat Service, Logstash, and OpenSearch Dashboards (ELK) with Amazon EC2 Linux, then your pipeline looks like this:Filebeat > Logstash > Amazon OpenSearch Service/Dashboards401 Unauthorized errorA 401 Unauthorized error from Logstash indicates that your OpenSearch Service domain is protected by fine-grained access control (FGAC) or Amazon Cognito. FGAC requires signed requests by a user or role, which must be defined in the domain's access policy. If you receive a 401 Unauthorized error, make sure that you properly turn on FGAC in your Logstash configuration file.For example:output {  elasticsearch {    hosts => ["https://vpc-examplestack-5crrfyysa2ratcl3ursmung33q.us-east-1.es.amazonaws.com:443"]    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"    ilm_enabled => false    user => "elastic"    password => "changeme"  }}403 Forbidden errorWhen you use and configure Logstash to send data to OpenSearch Service, you might receive a 403 Forbidden error. This error occurs when Logstash doesn't have necessary permissions and fails to verify your AWS Identity Access Management (IAM) identity. To resolve this issue, make sure to sign your requests to OpenSearch Service using your IAM credentials.To sign OpenSearch Service requests using Logstash, follow these steps:1.    Install the Logstash plugin for OpenSearch Service:bin/logstash-plugin install logstash-output-amazon_es2.     Attach an IAM role to the Amazon EC2 instance, like this:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": [        "es:ESHttp*"      ],      "Resource": "[Amazon-OpenSearch-Domain-ARN]"    }  ]}3.    Update your Logstash configuration settings to use the "amazon_es" Logstash plugin as the output in your pipeline:output {  amazon_es {    hosts => ["domain-endpoint"]    ssl => true    region => "us-east-1"    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"  }}Logstash x-pack installation errorIf you get errors with x-pack when you start up Logstash, manually turn off the x-pack plugin from your registry file.To manually turn off the x-pack plugin, follow these steps:1.    Open the following file:/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb2.    Find load_xpack and comment inline:"load_xpack unless LogStash::OSS" >> "#load_xpack unless LogStash::OSS"Note: You can check your configuration files to confirm whether the Index Life Management (ILM) settings (ilm.enabled and ilm_enabled) are both set to "false." Turning off these ILM settings in your configuration files eliminates startup errors for the x-pack plugin.Related informationLoading data into Amazon OpenSearch Service with LogstashFollow"