How do I troubleshoot high JVM memory pressure on my OpenSearch Service cluster?
"My Amazon OpenSearch Service cluster has high JVM memory pressure, and I don't know how to resolve this."
"My Amazon OpenSearch Service cluster has high JVM memory pressure, and I don't know how to resolve this.Short descriptionBy default, OpenSearch Service uses 50% of an instance's RAM for JVM heaps up to 32 GiB in size. The JVM memory pressure specifies the percentage of the Java heap in a cluster node. The following guidelines indicate what the JVM memory pressure percentages mean:If JVM memory pressure reaches 75%, then OpenSearch Service initiates the Concurrent Mark Sweep (CMS) garbage collector for x86 Amazon Elastic Compute Cloud (Amazon EC2) instance types. ARM-based Graviton Amazon EC2 instance types use the Garbage-First (G1) garbage collector that uses additional short pauses and heap defragmentation. The garbage collection is a CPU-intensive process. If memory usage continues to grow, then you might encounter ClusterBlockException, JVM OutOfMemoryError, or other cluster performance issues. For more information, see Recovering from a continuous heavy processing load.If JVM memory pressure exceeds 92% for 30 minutes, then OpenSearch Service blocks all write operations.If JVM memory pressure reaches 100%, then OpenSearch Service JVM is configured to exit and eventually restarts on OutOfMemory (OOM).The following reasons can cause high JVM memory pressure:Spikes in the numbers of requests to the cluster.Aggregations, wildcards, and selecting wide time ranges in the queries.Unbalanced shard allocations across nodes or too many shards in a cluster.Field data or index mapping explosions.Instance types that can't handle incoming loads.ResolutionReduce traffic to the cluster to resolve high JVM memory pressure issues. To reduce traffic to the cluster, follow these best practices:Clear the field data cache with the POST /index_name/_cache/clear?fielddata=true API operation.Note: Clearing the cache can disrupt queries that are in progress.Don't aggregate text fields or change the mapping type to "keyword".Scale the domain so that the maximum heap size per node is 32 GB.Turn on slow logs (OpenSearch website) to determine faulty requests.Note: Verify that the JVM memory pressure is below 90%. For more information about slow Elasticsearch queries, see Advanced tuning: finding and fixing slow Elasticsearch queries on the Elasticsearch website.Choose the correct number of shards to optimize search or indexing. For more information about indexing and shard count, see How do I rebalance the uneven shard distribution in my Amazon OpenSearch Service cluster?Delete old or unused indices to reduce the number of shards.For advanced users, you can update the parent field data cache allocation, or request circuit breaker settings according to your use case. For more information about JVM circuit breakers, see JVM OutOfMemoryError.For more information about how to troubleshoot high JVM memory pressure, see Why did my OpenSearch Service node crash?Related informationTroubleshooting Amazon OpenSearch ServiceHow can I scale up or scale out an Amazon OpenSearch Service domain?Get started with Amazon Elasticsearch Service: How many shards do I need?Follow"