Spark Saving Results to HDFS
<p>I am facing strange behavior while saving my parsed XML file using spark with one master and three worker nodes on HDFS, the problem is </p>

<p>when i parsed XMLFile and trying to save in HDFS then file is not able to save with all the parsed result.</p>

<p>and when i executed same code with local-Mode by specifying </p>

<pre><code>sc = SparkContext("local", "parser") 

and the spark-submit will be ./bin/spark-submit xml_parser.py
</code></pre>

<p>this run gives 117mb parsed file on hdfs with complete records.</p>

<p>And in case of executing code in spark-client mode then i did the following,</p>

<pre><code> sc = SparkContext("spark://master:7077", "parser") 
</code></pre>

<p>And the spark submit is,</p>

<pre><code>./bin/spark-submit --master yarn-client --deploy-mode client --driver-memory 7g --executor-memory 4g  --executor-cores 2  xml_parser.py 1000
</code></pre>

<p>giving me 19mb file on hdfs with incomplete records.</p>

<p>for saving the result in both the cases i am using rdd.saveAsTextFile("hdfs://")</p>

<p>i am using spark1.6.1-hadoop2.6 
and Apache hadoop 2.7.2</p>

<p>can anyone help me out. I am not getting why it is happening.
I have the following sparkCluster,</p>

<p>1-master 8GbRAM</p>

<p>2-workerNode1 8GbRAM</p>

<p>3-WorkerNode2 8GbRAM</p>

<p>4-workerNode3 8GbRAM</p>

<p>and i have configured above cluster over Hadoop-2.7.2 with 1 master and 3 DataNode,</p>

<p>if I jps On severNode gives me,</p>

<p>24097 Master</p>

<p>21652 Jps</p>

<p>23398 NameNode</p>

<p>23799 ResourceManager</p>

<p>23630 SecondaryNameNode</p>

<p>the JPS On all the DataNodes,</p>

<p>8006 Worker</p>

<p>7819 NodeManager</p>

<p>27164 Jps</p>

<p>7678 DataNode</p>

<p>By checking HadoopNameNode ui master:9000 give me the three live DataNodes</p>

<p>by checking SparkMaster Ui  on master:7077 gives me three live workers </p>

<p>please have look here,</p>

<pre><code>sc = SpakContext("spark://master:7077", "parser")
--------------------------------------------
 contains the logic of XMLParsing
--------------------------------------------
and I am appending the result in one list like,
cc_list.append([final_cll, Date,Time,int(cont[i]), float(values[i]),0])
Now I am Parallelizing the above cc_list like
 parallel_list = sc.parallelize(cc_list)
 parallel_list.saveAsTextFile("hdfs://master:9000/ some path")
 Now I am Doing some operations here.
 new_list = sc.textFile("hdfs://localhost:9000/some path/part-00000).map(lambda line:line.split(','))

 result = new_list.map(lambda x: (x[0]+',   '+x[3],float(x[4]))).sortByKey('true').coalesce(1)
 result = result.map(lambda x:x[0]+','+str(x[1]))
 result = result.map(lambda x: x.lstrip('[').rstrip(']').replace(' ','')).saveAsTextFile("hdfs://master:9000/some path1))
</code></pre>