How do I resolve the errors "session '0' not found" and "Error sending http request and maximum retry encountered" in Amazon EMR?
I want to troubleshoot errors when I run an Apache Livy application using a Jupyter Notebook on Amazon EMR.
"I want to troubleshoot errors when I run an Apache Livy application using a Jupyter Notebook on Amazon EMR.Short descriptionYou get one of the following errors when you run an Apache Livy application using a Jupyter Notebook on Amazon EMR:'404' from xxxxxxxx with error payload: "session '0' not found""The code failed because of a fatal error: Error sending http request and maximum retry encountered."These errors usually happen when you keep the Jupyter Notebook session running until it times out. To resolve these errors, increase the value of the livy.server.session.timeout property in /etc/livy/conf/livy.conf on the master node. Then, restart livy-server.ResolutionYou can modify livy.server.session.timeout on a running Amazon EMR cluster or when launching a new cluster.On a running cluster1.    Open /etc/livy/conf/livy.conf on the master node, and then modify the livy.server.session.timeout value (the default value is 1h). Example:sudo vim /etc/livy/conf/livy.conflivy.server.session.timeout 2h2.    Run the following commands on the master node to restart livy-server:Suppose that you're using one of the following Amazon EMR release versions that's based on Amazon Linux 2:Amazon EMR release version 5.30.0 or laterAmazon EMR 6.x seriesThen, run the following commands:sudo systemctl stop livy-serversudo systemctl start livy-serverIf you're using the Amazon EMR release version 5.29.0 or earlier, then run the following commands:sudo stop livy-serversudo start livy-serverWhen livy-server is restarting, your cluster is unavailable. To avoid downtime, configure the Livy application when you launch an Amazon EMR cluster.On a new clusterAdd a configuration objectsimilar to the following when you launch a cluster using Amazon EMR release version 4.6.0 or later. In addition to modifying livy.server.session.timeout, you can modify the following related properties, if desired:livy.server.session.timeout-check: When turned on, Livy stops idle sessions that reach the timeout threshold. The default setting is true.livy.server.yarn.app-lookup-timeout: The duration that Livy looks for the YARN application before considering it lost. The default setting is 60s.[    {        "Classification": "livy-conf",        "Properties": {            "livy.server.session.timeout-check": "true",            "livy.server.session.timeout": "2h",            "livy.server.yarn.app-lookup-timeout": "120s"        }    }]After you're done running the job, be sure to close the session in Jupyter or Zeppelin. When too many sessions are open, new jobs can't start until resources become available.Related informationApache LivyJupyter Notebook on Amazon EMRFollow"