Can't connect pyspark to master
<p>I have setup a three-node spark cluster that is also used as a hadoop cluster.<br>
master/worker1 is also namenode/datanode1<br>
worker2 is also datanode2<br>
worker3 is also datanode3<br></p>

<p>The nodes are VMs with private ip addresses but i also created a static ip address for them.</p>

<p>private ip: 192.168.0.4  -  static ip: x.x.x.117<br>
private ip: 192.168.0.7  -  static ip: x.x.x.118<br>
private ip: 192.168.0.2  -  static ip: x.x.x.120<br></p>

<p>Hadoop Version is hadoop-2.6.3<br>
Spark version is spark-1.5.2-bin-hadoop2.6<br>
Java version is 1.7.0_79<br>
<br>
When I used the command-line:<br>
<code>$ MASTER=spark://x.x.x.117:7077 pyspark --master yarn-client</code><br><br>
it does not give any errors and after all the verbose messages it displays on screen - i would eventually get the pyspark prompt and I can run spark job.  It just it's only running locally.
<br>
Also  when I check the Spark  WebUI:  <a href="http://x.x.x.117:8080" rel="nofollow">http://x.x.x.117:8080</a>, the pyspark application does not show up under the "Running Applications" section of the page.    I suspect that the pyspark shell is not really running in cluster mode.
<br><br>
So I tried this following command:<br>
<code>$ MASTER=spark://x.x.x.117:7077 pyspark</code><br><br>
The above command gives these messages on the console:<br><br></p>

<pre>
Python 2.7.5 (default, Jun 24 2015, 00:41:19)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
16/01/06 20:14:39 INFO spark.SparkContext: Running Spark version 1.5.2
16/01/06 20:14:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/01/06 20:14:40 INFO spark.SecurityManager: Changing view acls to: centos
16/01/06 20:14:40 INFO spark.SecurityManager: Changing modify acls to: centos
16/01/06 20:14:40 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(centos); users with modify permissions: Set(centos)
16/01/06 20:14:41 INFO slf4j.Slf4jLogger: Slf4jLogger started
16/01/06 20:14:41 INFO Remoting: Starting remoting
16/01/06 20:14:41 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.0.4:51079]
16/01/06 20:14:41 INFO util.Utils: Successfully started service 'sparkDriver' on port 51079.
16/01/06 20:14:41 INFO spark.SparkEnv: Registering MapOutputTracker
16/01/06 20:14:41 INFO spark.SparkEnv: Registering BlockManagerMaster
16/01/06 20:14:41 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-0d984b69-ad9c-4ced-ae65-ffd3bc1c79f5
16/01/06 20:14:41 INFO storage.MemoryStore: MemoryStore started with capacity 2.6 GB
16/01/06 20:14:41 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-f3b76c61-812b-4413-b86e-f42c1399d548/httpd-3a2c827d-4d6c-4d2e-8625-db27851c143d
16/01/06 20:14:41 INFO spark.HttpServer: Starting HTTP Server
16/01/06 20:14:41 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/01/06 20:14:41 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:42438
16/01/06 20:14:41 INFO util.Utils: Successfully started service 'HTTP file server' on port 42438.
16/01/06 20:14:41 INFO spark.SparkEnv: Registering OutputCommitCoordinator
16/01/06 20:14:41 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/01/06 20:14:41 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
16/01/06 20:14:41 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
16/01/06 20:14:41 INFO ui.SparkUI: Started SparkUI at http://192.168.0.4:4040
16/01/06 20:14:41 WARN metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
16/01/06 20:14:42 INFO client.AppClient$ClientEndpoint: Connecting to master spark://x.x.x.117:7077...
16/01/06 20:15:02 ERROR util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[appclient-registration-retry-thread,5,main]
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2d65e4c4 rejected from java.util.concurrent.ThreadPoolExecutor@7c8e1724[Running, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
        at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1.apply(AppClient.scala:96)
        at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1.apply(AppClient.scala:95)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
        at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
        at org.apache.spark.deploy.client.AppClient$ClientEndpoint.tryRegisterAllMasters(AppClient.scala:95)
        at org.apache.spark.deploy.client.AppClient$ClientEndpoint.org$apache$spark$deploy$client$AppClient$ClientEndpoint$$registerWithMaster(AppClient.scala:121)
        at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2$$anonfun$run$1.apply$mcV$sp(AppClient.scala:132)
        at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1119)
        at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2.run(AppClient.scala:124)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
16/01/06 20:15:02 INFO storage.DiskBlockManager: Shutdown hook called
16/01/06 20:15:02 INFO util.ShutdownHookManager: Shutdown hook called
16/01/06 20:15:02 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-f3b76c61-812b-4413-b86e-f42c1399d548
Traceback (most recent call last):
  File "/opt/spark-1.5.2-bin-hadoop2.6/python/pyspark/shell.py", line 43, in 
    sc = SparkContext(pyFiles=add_files)
  File "/opt/spark-1.5.2-bin-hadoop2.6/python/pyspark/context.py", line 113, in __init__
    conf, jsc, profiler_cls)
  File "/opt/spark-1.5.2-bin-hadoop2.6/python/pyspark/context.py", line 170, in _do_init
    self._jsc = jsc or self._initialize_context(self._conf._jconf)
  File "/opt/spark-1.5.2-bin-hadoop2.6/python/pyspark/context.py", line 224, in _initialize_context
    return self._jvm.JavaSparkContext(jconf)
  File "/opt/spark-1.5.2-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 699, in __call__
  File "/opt/spark-1.5.2-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 369, in send_command
  File "/opt/spark-1.5.2-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 362, in send_command
  File "/opt/spark-1.5.2-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 318, in _get_connection
  File "/opt/spark-1.5.2-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 325, in _create_connection
  File "/opt/spark-1.5.2-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 432, in start
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server
>>>
</pre>

<p>Looking at the log files generated for the master:<br></p>

<pre>
16/01/06 19:32:30 INFO master.Master: Registered signal handlers for [TERM, HUP, INT]
16/01/06 19:32:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classe
s where applicable
16/01/06 19:32:31 INFO spark.SecurityManager: Changing view acls to: root
16/01/06 19:32:31 INFO spark.SecurityManager: Changing modify acls to: root
16/01/06 19:32:31 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permiss
ions: Set(root); users with modify permissions: Set(root)
16/01/06 19:32:31 INFO slf4j.Slf4jLogger: Slf4jLogger started
16/01/06 19:32:31 INFO Remoting: Starting remoting
16/01/06 19:32:32 INFO util.Utils: Successfully started service 'sparkMaster' on port 7077.
16/01/06 19:32:32 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkMaster@cassandra-spark-1:7077]
16/01/06 19:32:32 INFO master.Master: Starting Spark master at spark://cassandra-spark-1:7077
16/01/06 19:32:32 INFO master.Master: Running Spark version 1.5.2
16/01/06 19:32:32 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/01/06 19:32:33 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:8080
16/01/06 19:32:33 INFO util.Utils: Successfully started service 'MasterUI' on port 8080.
16/01/06 19:32:33 INFO ui.MasterWebUI: Started MasterWebUI at http://192.168.0.4:8080
16/01/06 19:32:33 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/01/06 19:32:33 INFO server.AbstractConnector: Started SelectChannelConnector@cassandra-spark-1:6066
16/01/06 19:32:33 INFO util.Utils: Successfully started service on port 6066.
16/01/06 19:32:33 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066
16/01/06 19:32:33 INFO master.Master: I have been elected leader! New state: ALIVE
16/01/06 19:32:35 INFO master.Master: Registering worker 192.168.0.7:52930 with 2 cores, 6.6 GB RAM
16/01/06 19:32:35 INFO master.Master: Registering worker 192.168.0.2:48119 with 2 cores, 6.6 GB RAM
16/01/06 19:32:35 INFO master.Master: Registering worker 192.168.0.4:56830 with 2 cores, 6.6 GB RAM
16/01/06 19:33:32 ERROR akka.ErrorMonitor: dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Act
or[akka.tcp://sparkMaster@x.x.x.x:7077/]] arriving at [akka.tcp://sparkMaster@x.x.x.117:7077] inbound addresses are [ak
ka.tcp://sparkMaster@cassandra-spark-1:7077]
akka.event.Logging$Error$NoCause$
16/01/06 19:33:52 INFO master.Master: 192.168.0.4:35598 got disassociated, removing it.
16/01/06 19:33:52 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkDriver@192.168.0.4:3559
8] has failed, address is now gated for [5000] ms. Reason: [Disassociated]
16/01/06 19:33:52 INFO master.Master: 192.168.0.4:35598 got disassociated, removing it.
16/01/06 19:38:36 ERROR akka.ErrorMonitor: dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Act
or[akka.tcp://sparkMaster@192.168.0.4:7077/]] arriving at [akka.tcp://sparkMaster@192.168.0.4:7077] inbound addresses are [akka.t
cp://sparkMaster@cassandra-spark-1:7077]
akka.event.Logging$Error$NoCause$
16/01/06 19:38:56 INFO master.Master: 192.168.0.4:36078 got disassociated, removing it.
16/01/06 19:38:56 INFO master.Master: 192.168.0.4:36078 got disassociated, removing it.
16/01/06 19:38:56 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkDriver@192.168.0.4:3607
8] has failed, address is now gated for [5000] ms. Reason: [Disassociated]
16/01/06 20:14:42 ERROR akka.ErrorMonitor: dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Act
or[akka.tcp://sparkMaster@x.x.x.117:7077/]] arriving at [akka.tcp://sparkMaster@x.x.x.117:7077] inbound addresses are [ak
ka.tcp://sparkMaster@cassandra-spark-1:7077]
akka.event.Logging$Error$NoCause$
16/01/06 20:15:02 INFO master.Master: 192.168.0.4:51079 got disassociated, removing it.
16/01/06 20:15:02 INFO master.Master: 192.168.0.4:51079 got disassociated, removing it.
16/01/06 20:15:02 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkDriver@192.168.0.4:5107
9] has failed, address is now gated for [5000] ms. Reason: [Disassociated]
</pre>

<p><br>
I would appreciate any help.  Thanks! <br></p>