executing spark from client (windows) on yarn cluster (linux): Error no sheme for Filesystem "C"
<p>I want to deploy a spark application (just a simple Hello World app) to my hadoop cluster.
Using spark submit on my windows machine I execute the application in client mode with --master yarn. The connection to the hadoop cluster is successful, as it can be seen in the log file on the cluster. (hadoop conf files have been downladed from the cluster and saved on the client windows machine, environment variables have been set).</p>

<p>Using hadoop 2.7 and spark 1.6</p>

<p>This is the spark-submit command used:</p>

<pre><code>&gt;spark-submit --master yarn --class "SimpleApp" ..\..\SimpleApp\target\scala-2.11\simple-project_2.11-1.0.jar --deploy-mode client
</code></pre>

<p>This is the error message returned:</p>

<pre><code>16/08/10 10:52:45 ERROR SparkContext: Error initializing SparkContext.
java.io.IOException: No FileSystem for scheme: C
</code></pre>

<p>The full error message:</p>

<pre><code>Warning: Ignoring non-spark config property: fs.hdfs.impl=org.apache.hadoop.hdfs.DistributedFileSystem
16/08/10 10:52:32 INFO SparkContext: Running Spark version 1.6.2
16/08/10 10:52:32 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/08/10 10:52:32 INFO SecurityManager: Changing view acls to: sensored,sensored
16/08/10 10:52:32 INFO SecurityManager: Changing modify acls to: sensored,sensored
16/08/10 10:52:32 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(sensored,sensored); users with modify permissions: Set(sensored,sensored)
16/08/10 10:52:33 INFO Utils: Successfully started service 'sparkDriver' on port 52340.
16/08/10 10:52:33 INFO Slf4jLogger: Slf4jLogger started
16/08/10 10:52:33 INFO Remoting: Starting remoting
16/08/10 10:52:34 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@sensored,sensored]
16/08/10 10:52:34 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 52353.
16/08/10 10:52:34 INFO SparkEnv: Registering MapOutputTracker
16/08/10 10:52:34 INFO SparkEnv: Registering BlockManagerMaster
16/08/10 10:52:34 INFO DiskBlockManager: Created local directory at C:\Users\sensored,sensored\AppData\Local\Temp\blockmgr-35b65f9a-47a3-4d1e-940f-be8bc9c78b8f
16/08/10 10:52:34 INFO MemoryStore: MemoryStore started with capacity 511.1 MB
16/08/10 10:52:34 INFO SparkEnv: Registering OutputCommitCoordinator
16/08/10 10:52:34 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/08/10 10:52:34 INFO SparkUI: Started SparkUI at http://10.61.156.198:4040
16/08/10 10:52:34 INFO HttpFileServer: HTTP File server directory is C:\Users\sensored,sensored\AppData\Local\Temp\spark-ec3f2bdc-e11e-48b5-a3f6-7d8ae97b8a5e\httpd-e39c2eda-b9aa-40df-95c9-eae44b53c4ee
16/08/10 10:52:34 INFO HttpServer: Starting HTTP Server
16/08/10 10:52:34 INFO Utils: Successfully started service 'HTTP file server' on port 52356.
16/08/10 10:52:34 INFO SparkContext: Added JAR file:/C:/Users/sensored,sensored/Documents/spark-1.6.2-bin-hadoop2.6/bin/../../SimpleApp/target/scala-2.11/simple-project_2.11-1.0.jar at http://sensored,sensored:52356/jars
/simple-project_2.11-1.0.jar with timestamp 1470819154823
16/08/10 10:52:35 INFO TimelineClientImpl: Timeline service address: http://sensored,sensored:8188/ws/v1/timeline/
16/08/10 10:52:35 INFO RMProxy: Connecting to ResourceManager at sensored,sensored/sensored,sensored.188:8050
16/08/10 10:52:36 INFO Client: Requesting a new application from cluster with 2 NodeManagers
16/08/10 10:52:36 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (221184 MB per container)
16/08/10 10:52:36 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
16/08/10 10:52:36 INFO Client: Setting up container launch context for our AM
16/08/10 10:52:36 INFO Client: Setting up the launch environment for our AM container
16/08/10 10:52:36 INFO Client: Preparing resources for our AM container
16/08/10 10:52:36 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
16/08/10 10:52:36 INFO Client: Uploading resource file:/C:/Users/sensored,sensored/Documents/spark-1.6.2-bin-hadoop2.6/lib/spark-assembly-1.6.2-hadoop2.6.0.jar -&gt; hdfs://sensored,sensored:8020/user/sensored,sensored/.sparkSt
aging/application_1468399950865_0375/spark-assembly-1.6.2-hadoop2.6.0.jar
16/08/10 10:52:41 INFO Client: Uploading resource file:/C:/Users/sensored,sensored/AppData/Local/Temp/spark-ec3f2bdc-e11e-48b5-a3f6-7d8ae97b8a5e/__spark_conf__7713144618669536696.zip -&gt; hdfs://sensored,sensored:80
20/user/sensored,sensored/.sparkStaging/application_1468399950865_0375/__spark_conf__7713144618669536696.zip
16/08/10 10:52:41 INFO SecurityManager: Changing view acls to: sensored,sensored
16/08/10 10:52:41 INFO SecurityManager: Changing modify acls to: sensored,sensored
16/08/10 10:52:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(sensored,sensored); users with modify permissions: Set(sensored,sensored)
16/08/10 10:52:41 INFO Client: Submitting application 375 to ResourceManager
16/08/10 10:52:41 INFO YarnClientImpl: Submitted application application_1468399950865_0375
16/08/10 10:52:42 INFO Client: Application report for application_1468399950865_0375 (state: ACCEPTED)
16/08/10 10:52:42 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1470819161610
         final status: UNDEFINED
         tracking URL: http://sensored,sensored:8088/proxy/application_1468399950865_0375/
         user: sensored,sensored
16/08/10 10:52:43 INFO Client: Application report for application_1468399950865_0375 (state: ACCEPTED)
16/08/10 10:52:44 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(null)
16/08/10 10:52:44 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -&gt; sensored,sensored, PROXY_URI_BASES -&gt; http://sensored,sensored:8088/proxy/application_1468399950865_0375), /proxy/application_1468399950865_0375
16/08/10 10:52:44 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
16/08/10 10:52:44 INFO Client: Application report for application_1468399950865_0375 (state: RUNNING)
16/08/10 10:52:44 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: 10.3.131.188
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1470819161610
         final status: UNDEFINED
         tracking URL: http://sensored,sensored:8088/proxy/application_1468399950865_0375/
         user: sensored,sensored
16/08/10 10:52:44 INFO YarnClientSchedulerBackend: Application application_1468399950865_0375 has started running.
16/08/10 10:52:44 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 52383.
16/08/10 10:52:44 INFO NettyBlockTransferService: Server created on 52383
16/08/10 10:52:44 INFO BlockManagerMaster: Trying to register BlockManager
16/08/10 10:52:44 INFO BlockManagerMasterEndpoint: Registering block manager 10.61.156.198:52383 with 511.1 MB RAM, BlockManagerId(driver, sensored,sensored, 52383)
16/08/10 10:52:44 INFO BlockManagerMaster: Registered BlockManager
16/08/10 10:52:45 ERROR SparkContext: Error initializing SparkContext.
java.io.IOException: No FileSystem for scheme: C
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
        at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1686)
        at org.apache.spark.scheduler.EventLoggingListener.&lt;init&gt;(EventLoggingListener.scala:66)
        at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:547)
        at SimpleApp$.main(SimpleApp.scala:10)
        at SimpleApp.main(SimpleApp.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/08/10 10:52:45 INFO SparkUI: Stopped Spark web UI at http://10.61.156.198:4040
16/08/10 10:52:45 INFO YarnClientSchedulerBackend: Shutting down all executors
16/08/10 10:52:45 INFO YarnClientSchedulerBackend: Interrupting monitor thread
16/08/10 10:52:45 INFO YarnClientSchedulerBackend: Asking each executor to shut down
16/08/10 10:52:45 INFO YarnClientSchedulerBackend: Stopped
16/08/10 10:52:45 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/08/10 10:52:45 INFO MemoryStore: MemoryStore cleared
16/08/10 10:52:45 INFO BlockManager: BlockManager stopped
16/08/10 10:52:45 INFO BlockManagerMaster: BlockManagerMaster stopped
16/08/10 10:52:45 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/08/10 10:52:45 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/08/10 10:52:45 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
16/08/10 10:52:45 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.io.IOException: No FileSystem for scheme: C
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
        at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1686)
        at org.apache.spark.scheduler.EventLoggingListener.&lt;init&gt;(EventLoggingListener.scala:66)
        at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:547)
        at SimpleApp$.main(SimpleApp.scala:10)
        at SimpleApp.main(SimpleApp.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/08/10 10:52:45 INFO ShutdownHookManager: Shutdown hook called
16/08/10 10:52:45 INFO ShutdownHookManager: Deleting directory C:\Users\sensored,sensored\AppData\Local\Temp\spark-ec3f2bdc-e11e-48b5-a3f6-7d8ae97b8a5e\httpd-e39c2eda-b9aa-40df-95c9-eae44b53c4ee
16/08/10 10:52:45 INFO ShutdownHookManager: Deleting directory C:\Users\sensored,sensored\AppData\Local\Temp\spark-ec3f2bdc-e11e-48b5-a3f6-7d8ae97b8a5e
</code></pre>