Is there any blueprint for a data-pipeline?
<p>I use Spark for data processing, but starting from the datasources (mostly csv files) I would like to put in place a data-pipeline which has right stages to control/test/manipulate data and deploy them to different "stages" (CI-CD/QA/UAT/LIVE/etc).</p>

<p>Is there any valid data-pipeline "blueprint" for it?</p>