Apache Spark: Which data storage and data format to choose
<p>I'm going to write a sales analytics application with Spark. Therefore I get a delta-dataset every night with new sales data (the sellings of the day before). Later I want to realize some analytics like Association-Rules or popularity of products. </p>

<p>The sales data contains information about:</p>

<ul>
<li>store-id</li>
<li>article-group</li>
<li>timestamp of cash-point</li>
<li>article GTIN</li>
<li>amount</li>
<li>price</li>
</ul>

<p>So far I used a simple <code>.textFile</code> method and RDDs in my Applications. I heard something about DataFrame and Parquet, which is a table-like data format for text files, right? And what about storing the data once in a database (I have HBase installed in a Hadoop cluster) and later read this? </p>

<p>Can someone give a short overview of the different types of save-/load-possibilities in Spark? And give a recommendation what to use for this data?</p>

<p>The data-amount are actually about 6 GB, which represent data data for 3 stores for about 1 year. Later I will work with data of ~500 stores and time-period of ~5 years.</p>