Spark Job submitted - Waiting (TaskSchedulerImpl : Initial job not accepted)
<p>API call made to submit the Job. Response states - It is Running</p>

<p>On Cluster UI - </p>

<blockquote>
  <p>Worker (slave) - worker-20160712083825-172.31.17.189-59433 is Alive</p>
  
  <p>Core 1 out of 2 used</p>
  
  <p>Memory 1Gb out of 6 used</p>
</blockquote>

<p>Running Application</p>

<blockquote>
  <p>app-20160713130056-0020 - Waiting since 5hrs</p>
  
  <p>Cores - unlimited</p>
</blockquote>

<p>Job Description of the Application</p>

<p>Active Stage</p>

<blockquote>
  <p>reduceByKey at /root/wordcount.py:23</p>
</blockquote>

<p>Pending Stage</p>

<blockquote>
  <p>takeOrdered at /root/wordcount.py:26</p>
</blockquote>

<p>Running Driver - </p>

<pre><code>stderr log page for driver-20160713130051-0025 

WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
</code></pre>

<p>According to <a href="https://stackoverflow.com/questions/38118572/initial-job-has-not-accepted-any-resources-check-your-cluster-ui-to-ensure-that">Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources</a>
Slaves haven't been started - Hence it doesn't have resources.</p>

<p>However in my case - Slave 1 is working</p>

<p>According to <a href="https://stackoverflow.com/questions/32651568/unable-to-execute-more-than-a-spark-job-initial-job-has-not-accepted-any-resour">Unable to Execute More than a spark Job &quot;Initial job has not accepted any resources&quot;</a>
I am using deploy-mode = cluster (not client) Since I have 1 master 1 slave and Submit API is being called via Postman / anywhere</p>

<p>Also the Cluster has available Cores, RAM, Memory - Still Job throws the error
as conveyed by the UI</p>

<p>According to <a href="https://stackoverflow.com/questions/29469462/taskschedulerimpl-initial-job-has-not-accepted-any-resources">TaskSchedulerImpl: Initial job has not accepted any resources;</a>
I assigned </p>

<pre><code>~/spark-1.5.0/conf/spark-env.sh
</code></pre>

<p>Spark Environment Variables</p>

<pre><code>SPARK_WORKER_INSTANCES=1
SPARK_WORKER_MEMORY=1000m
SPARK_WORKER_CORES=2
</code></pre>

<p>Replicated those across the Slaves </p>

<pre><code>sudo /root/spark-ec2/copy-dir /root/spark/conf/spark-env.sh
</code></pre>

<p>All the cases in the answer to above question - were applicable still no solution found. Hence because I was working with APIs and Apache SPark - maybe some other assistance is required. </p>

<blockquote>
  <p>Edited July 18,2016</p>
  
  <p>Wordcount.py - My PySpark application code -</p>
</blockquote>

<pre><code>from pyspark import SparkContext, SparkConf

logFile = "/user/root/In/a.txt"

conf = (SparkConf().set("num-executors", "1"))

sc = SparkContext(master = "spark://ec2-54-209-108-127.compute-1.amazonaws.com:7077", appName = "MyApp", conf = conf)
print("in here")
lines = sc.textFile(logFile)
print("text read")
c = lines.count()
print("lines counted")
</code></pre>

<p>Error</p>

<pre><code>Starting job: count at /root/wordcount.py:11
16/07/18 07:46:39 INFO scheduler.DAGScheduler: Got job 0 (count at /root/wordcount.py:11) with 2 output partitions
16/07/18 07:46:39 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (count at /root/wordcount.py:11)
16/07/18 07:46:39 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/18 07:46:39 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/18 07:46:39 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (PythonRDD[2] at count at /root/wordcount.py:11), which has no missing parents
16/07/18 07:46:39 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 5.6 KB, free 56.2 KB)
16/07/18 07:46:39 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.4 KB, free 59.7 KB)
16/07/18 07:46:39 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on 172.31.17.189:43684 (size: 3.4 KB, free: 511.5 MB)
16/07/18 07:46:39 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
16/07/18 07:46:39 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (PythonRDD[2] at count at /root/wordcount.py:11)
16/07/18 07:46:39 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
16/07/18 07:46:54 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
</code></pre>

<p>According to <a href="https://stackoverflow.com/questions/35169015/spark-ui-showing-0-cores-even-when-setting-cores-in-app">Spark UI showing 0 cores even when setting cores in App</a>, </p>

<p><strong>Spark WebUI states zero cores used and indefinite wait no tasks running. The application is also using NO MEMORY whatsoever during run time or cores and immediately hits a status of waiting when starting</strong></p>

<p>Spark version 1.6.1
Ubuntu
Amazon EC2</p>