How do I monitor my Amazon OpenSearch Service cluster using CloudWatch alarms?
I want to monitor my Amazon OpenSearch Service cluster for stability issues. How can I effectively monitor my cluster?
"I want to monitor my Amazon OpenSearch Service cluster for stability issues. How can I effectively monitor my cluster?ResolutionImportant: Different versions of Elasticsearch use different thread pools to process calls to the _index API.Elasticsearch versions 1.5 and 2.3 use the index thread pool.Elasticsearch versions 5.x, 6.0, and 6.2 use the bulk thread pool. (Currently, the OpenSearch Service console doesn't include a graph for the bulk thread pool.)Elasticsearch versions 6.3 and later use the write thread pool.To monitor the health of your OpenSearch Service cluster, set the recommended Amazon CloudWatch alarms and the following OpenSearch Service cluster metric alarms:MasterReachableFromNodeKibanaHealthyNodesDiskQueueDepthThreadpoolIndexQueueThreadpoolSearchQueueYou can configure your OpenSearch Service metric alarms like this:MasterReachableFromNode:Statistic = MaximumValue = ‘=0’Frequency = 1 periodPeriod = 1 minuteIssue: Leader node is down.KibanaHealthyNodes:Statistic = AverageValue = ‘=0’Frequency = 1 periodPeriod = 1 minuteIssue: Indicates that the kibana index is unhealthy.DiskQueueDepth:Statistic = AverageValue = ‘>=100'Frequency = 1 periodPeriod = 5 minutesIssue: Disk Queue Depth is the number of I/O requests that are queued at a time against the storage. This could indicate a surge in requests or Amazon EBS throttling, resulting in increased latency.ThreadpoolIndexQueue and ThreadpoolSearchQueue:Statistic = MaximumValue = ‘>=20’Frequency = 1 periodPeriod = 1 minuteIssue: Indicates that there are requests getting queued up, which can be rejected. To verify the request status, check the CPU Utilization and Threadpool Index or Search rejects.To set up an Amazon CloudWatch alarm for your OpenSearch Service cluster, perform the following steps:1.    Open the Amazon CloudWatch console.2.    Go to the Alarm tab.3.    Choose Create Alarm.4.    Choose Select Metric.5.    Choose ES for your metric.6.    Select Per-Domain and Per-Client Metrics.7.    Select a metric and choose Next.8.    Configure the following settings for your Amazon CloudWatch alarm:Statistic = MaximumPeriod to 1 minuteThreshold type = StaticAlarm condition = Greater than or equal toThreshold value = 19.    Choose the Additional configuration tab.10.    Update the following configuration settings:Datapoints to alarm = Frequency stated aboveMissing data treatment = Treat missing data as ignore (maintain the alarm state)11.    Choose Next.12.    Choose the action that you want your alarm to take, and choose Next.13.    Set a name for your alarm, and then choose Next.14.    Choose Create Alarm.Note: If the alarm is triggered for CPUUtilization or JVMMemoryPressure, check your Amazon CloudWatch metrics to see if there's a spike coinciding with incoming requests. In particular, monitor these Amazon CloudWatch metrics: IndexingRate, SearchRate, and OpenSearchRequests.Related informationClusterBlockExceptionUsing Amazon CloudWatch alarmsFollow"