How can I review NUMA statistics for my workload running on EC2 instances?
I'm running my workload on Amazon Elastic Compute Cloud (Amazon EC2) Nitro-based instances. I want to review performance and non-uniform memory access (NUMA) statistics.
"I'm running my workload on Amazon Elastic Compute Cloud (Amazon EC2) Nitro-based instances. I want to review performance and non-uniform memory access (NUMA) statistics.Short descriptionThe following factors affect the performance of an application that's hosted on an EC2 instance:CPU over utilizationMemory useThe EBS volumeNetwork statisticsNot NUMA awareIn NUMA architecture, each CPU has access to its own assigned memory, known as local memory. Each CPU can also access memory that's allocated to other CPUs, known as foreign memory. If applications hosted on your instances aren't NUMA aware, then accessing the foreign memory incurs additional costs and might affect performance. Contact the application vendor to check if your application is NUMA aware.ResolutionReview NUMA statisticsNote: To review performance components other than NUMA statistics, see How do I troubleshoot slow connections to a website hosted on my EC2 instance?To review NUMA statistics, complete the following steps:1.    Run the following command to check if the instance type is in a single NUMA or multi-NUMA node.Note: The instance type is r5.16xlarge has 2 NUMA nodes.lscpu | grep -i numaNUMA node(s): 22.    Run the following command to install the numactl package:sudo yum install numactl 3.    Run the following command to review the NUMA topology:sudo numactl -HIn the following example output the NUMA topology is divided into two nodes, node 0 and node 1. Node 0 has 32 CPUs and 255,225 MB of memory assigned to it. Node 1 has the other half of the CPUs and 254,924 MB of assigned memory. The node distance indicates the latencies in accessing memory pages from the other node.available: 2 nodes (0-1)node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47node 0 size: 255140 MBnode 0 free: 254794 MBnode 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63node 1 size: 255225 MBnode 1 free: 254924 MBnode distances:node   0   10:  10  211:  21  104.    Run the following command to review the NUMA statistics:sudo numastatThe following is an example of the command output:node                       node0           node1numa_hit                  314825          288025numa_miss                      0               0numa_foreign                   0               0interleave_hit             37958           37620local_node                311752          248476other_node                  3073           39549For a detailed explanation of each statistic, see A.11. NUMASTAT on the Red Hat Customer Portal.Increases in the numa_foreign and numa_miss statistics might indicate that the application running on the EC2 instance isn't NUMA aware. This might affect performance. To resolve this issue, turn off NUMA at the grub level. Or, try binding the application to the specific NUMA node. For more information, see Operating system optimizations.Permanently turn off NUMA To permanently turn off NUMA on Amazon Linux 2, CentOS7, or RHEL 7, run the following commands:sudo vi /etc/default/grubGRUB_CMDLINE_LINUX_DEFAULT="numa=off"sudo grub2-mkconfig -o /etc/grub2.cfgsudo rebootFollow"