SPARK Reading CSV from FTP: Input path does not exist
<p>I'm trying to do something that is supposed to be quite straightforward, but can't manage to do it.</p>

<p>I have a .csv file in a FTP server from a customer. The path is something as such:</p>

<p><code>ftp://[user]:[passwd]@[IP-ADDRESS]/file.csv</code></p>

<p>By copy and pasting the address, I can easily access the file on my browser (or any other program). But I can't access it on pyspark. </p>

<p>Here is what I'm trying to do, using databricks spark-csv (<a href="https://github.com/databricks/spark-csv" rel="nofollow">https://github.com/databricks/spark-csv</a>):</p>

<pre><code>file_path = ftp://[user]:[passwd]@[IP-ADDRESS]/file.csv
reader = sqlContext.read.format('com.databricks.spark.csv')
         .options(header=False, charset='cp860', inferschema='true', delim=";")
         .load(file_path)
</code></pre>

<p>and I receive the following error:</p>

<pre><code>org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: ftp://[user]:[passwd]@[IP-ADDRESS]/file.csv
</code></pre>

<p>Trying to read file as text file:</p>

<pre><code>df = sqlContext.read.text("ftp://[user]:[passwd]@[IP-ADDRESS]/SALES_37_TIRADENTES_2016-09-01.csv", )
df.collect()
</code></pre>

<p>error:</p>

<pre><code>Py4JJavaError: An error occurred while calling o147.collectToPython.
: java.io.IOException: No input paths specified in job
</code></pre>

<p>I'm pretty sure this has something to do with FTP access, but I'm not sure.</p>

<hr>

<h2>EDIT</h2>

<p>I ended up downloading the file using the Python's <code>ftplib</code> and loading the file into the RDD. Probably a suboptimal approach, if the size of your data is way too big, but it works.</p>