How to decrease heartbeat time of slave nodes in Hadoop
<p>I am working on AWS EMR. </p>

<p>I want to get the information of died task node as soon as possible. But as per default setting in hadoop, heartbeat is shared after every 10 minutes.</p>

<p>This is the default key-value pair in mapred-default - <a href="https://hadoop.apache.org/docs/r2.7.1/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapreduce.jobtracker.expire.trackers.interval  :  600000ms</a></p>

<p>I tried to modify default value to 6000ms using - <a href="http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-configure-apps.html">this link</a></p>

<p>After that, whenever I terminate any ec2 machine from EMR cluster, I am not able to see state change that fast.(in 6 seconds)</p>

<p>Resource manager REST API - <strong><a href="http://MASTER_DNS_NAME:8088/ws/v1/cluster/nodes">http://MASTER_DNS_NAME:8088/ws/v1/cluster/nodes</a></strong></p>

<p><strong>Questions-</strong></p>

<ol>
<li>What is the command to check the mapreduce.jobtracker.expire.trackers.interval value in running EMR cluster(Hadoop cluster)?</li>
<li>Is this the right key I am using to get the state change ? If it is not, please suggest any other solution.</li>
<li>What is the difference between DECOMMISSIONING vs DECOMMISSIONED vs LOST state of nodes in Resource manager UI ?</li>
</ol>

<p><strong>Update</strong></p>

<p>I tried numbers of times, but it is showing ambiguous behaviour. Sometimes, it moved to DECOMMISSIONING/DECOMMISIONED state, and sometime it directly move to LOST state after 10 minutes.</p>

<p>I need a quick state change, so that I can trigger some event.</p>

<p>Here is my sample code -</p>

<pre><code>List&lt;Configuration&gt; configurations = new ArrayList&lt;Configuration&gt;();

        Configuration mapredSiteConfiguration = new Configuration();
        mapredSiteConfiguration.setClassification("mapred-site");
        Map&lt;String, String&gt; mapredSiteConfigurationMapper = new HashMap&lt;String, String&gt;();
        mapredSiteConfigurationMapper.put("mapreduce.jobtracker.expire.trackers.interval", "7000");
        mapredSiteConfiguration.setProperties(mapredSiteConfigurationMapper);

        Configuration hdfsSiteConfiguration = new Configuration();
        hdfsSiteConfiguration.setClassification("hdfs-site");
        Map&lt;String, String&gt; hdfsSiteConfigurationMapper = new HashMap&lt;String, String&gt;();
        hdfsSiteConfigurationMapper.put("dfs.namenode.decommission.interval", "10");
        hdfsSiteConfiguration.setProperties(hdfsSiteConfigurationMapper);

        Configuration yarnSiteConfiguration = new Configuration();
        yarnSiteConfiguration.setClassification("yarn-site");
        Map&lt;String, String&gt; yarnSiteConfigurationMapper = new HashMap&lt;String, String&gt;();
        yarnSiteConfigurationMapper.put("yarn.resourcemanager.nodemanagers.heartbeat-interval-ms", "5000");
        yarnSiteConfiguration.setProperties(yarnSiteConfigurationMapper);

        configurations.add(mapredSiteConfiguration);
        configurations.add(hdfsSiteConfiguration);
        configurations.add(yarnSiteConfiguration);
</code></pre>

<p>This is the settings that I changed into AWS EMR (internally Hadoop) to reduce the time between state change from RUNNING to other state(DECOMMISSIONING/DECOMMISIONED/LOST).</p>