How do I check the resource utilization for my SageMaker notebook instance?
"I started an Amazon SageMaker notebook instance to train models, or to load large datasets, and the notebook instance appears to be frozen. How do I view my SageMaker instance resource use?"
"I started an Amazon SageMaker notebook instance to train models, or to load large datasets, and the notebook instance appears to be frozen. How do I view my SageMaker instance resource use?ResolutionWhen using the SageMaker notebook instance resources to prototype, train models, or when working with large datasets, SageMaker’s browser or notebook instances might appear unresponsive. If your browser or notebook instances appear unresponsive, you can view your SageMaker resource utilization to view current resource use.You can view your SageMaker resource utilization using one of the following approaches:Running Linux-based commandsReviewing Amazon CloudWatch metricsViewing SageMaker resource utilization with Linux commandsSageMaker notebook instances are based on Amazon Linux. You can run Linux commands from the SageMaker terminal to view SageMaker resource utilization.To run SageMaker Linux commands to view your resource utilization, do the following:Open the SageMaker console.On the navigation pane, choose Notebook Instances.Open Jupyter or JupyterLab next to the SageMaker notebook instance of your choice.Open terminal.Run the following commands to view your SageMaker resource utilization:topThe preceding command displays available system memory (RAM) and processor load.ps -axThe preceding command displays tasks running and processor load.df -hThe preceding command displays disk space utilization and availability.free -mThe preceding command displays system memory (RAM) utilization and availability.Viewing SageMaker resource utilization using CloudWatchYou can use CloudWatch to view your SageMaker resource utilization by using a lifecycle configuration script. For example, the publish-instance-metrics script publishes the system-level metrics from the notebook instance into CloudWatch.To configure your SageMaker notebook instance to view all metrics from CloudWatch:Open the SageMaker console.On the navigation pane, choose Notebook Instances.Choose Open Jupyter or Jupyterlab next to the SageMaker notebook instance of your choice.Open terminal.Input the following command to open the amazon-cloudwatch-agent-config-wizard:sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizardFollow the steps in the wizard. When prompted, do the following:Choose On-premises hostChoose no for StatsD DaemonChoose no for CollectDWhen the wizard completes, it automatically creates a config.json file. This file is used in the next step.Start the CloudWatch agent on your server with the following command:sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:///opt/aws/amazon-cloudwatch-agent/bin/config.json -sFrom your CloudWatch console, choose Metrics, then choose CWAgent.The CWAgent displays your current SageMaker metrics.For more information about example AWS lifecycle configuration scripts for SageMaker notebooks, see amazon-sagemaker-notebook-instance-lifecycle-config-samples.Related informationMonitor Amazon SageMaker with Amazon CloudWatchMetrics collected by the CloudWatch agentMonitor Amazon SageMakerTerminals - Jupyter Project documentation for TerminalsFollow"