Akka Singleton Cluster: resolveOnce of a worker by master fails after restart
<p>I am using Akka Cluster 2.4.3 and trying to setup a simple cluster in my machine to understand its working better. I have a singleton cluster with remoting enabled with primary and standby master and one worker node. Each of these 3 run in separate JVMs</p>

<p>Things work fine when all the nodes are started the first time. If I kill and restart the worker, I see following issues happening</p>

<p><strong>Restart Worker</strong></p>

<p>When the worker comes back after restart, the master on receiving <code>MemberUp</code> event tries to resolve for the actorRef from the member address the following way</p>

<p><code>context.actorSelection(member.address.toString).resolveOne(15 seconds)</code></p>

<p>This fails with an exception saying <code>ActorNotFound</code>. This works with no problem when all the nodes are coming up for the first time in the cluster.</p>

<p><strong>Restart worker again</strong></p>

<p>This time, the worker comes up with the following message</p>

<p><code>
[WARN] [04/15/2016 18:24:24.991] [clustersystem-akka.remote.default-remote-dispatcher-5] [akka.remote.Remoting] Tried to associate with unreachable remote address [akka.tcp://clustersystem@host1:2551]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.]
</code></p>

<p><strong>Restart worker again</strong></p>

<p>This time the <code>resolveOne</code> on a <code>MemberUp</code> event works.</p>

<p>I am having a bit of difficulty in understanding what is happening here, I have looked into the docs but I did not find anything that will help me in there.</p>

<p><strong>application.conf</strong></p>

<pre><code>akka {
  actor {
    provider = "akka.cluster.ClusterActorRefProvider"
  }

  remote {
    enabled-transports = ["akka.remote.netty.tcp"]
  }

  log-dead-letters = off
  jvm-exit-on-fatal-error = on

  loglevel = "DEBUG"

  remote {
    log-remote-lifecycle-events = off
    netty.tcp {
      hostname = "host1"
      port = 0
    }
  }

  cluster {
    seed-nodes = [
      "akka.tcp://clustersystem@host1:2551",
      "akka.tcp://clustersystem@host1:2552"]

    auto-down-unreachable-after = 10s
  }


  extensions = ["akka.cluster.metrics.ClusterMetricsExtension"]
}
</code></pre>

<p>I start master nodes at ports 2551 and 2552 (provide the ports as command line args) and I start the worker on port 3551</p>