How do you automate pyspark jobs on emr using boto3 (or otherwise)?
<p>I am creating a job to parse massive amounts of server data, and then upload it into a <code>Redshift</code> database.</p>

<p>My job flow is as follows:</p>

<ul>
<li>Grab the log data from S3</li>
<li>Either use spark <code>dataframes</code> or spark sql to parse the data and write back out to S3</li>
<li>Upload the data from S3 to Redshift.</li>
</ul>

<p>I'm getting hung up on how to automate this though so that my process spins up an EMR cluster, bootstraps the correct programs for installation, and runs my python script that will contain the code for parsing and writing.  </p>

<p>Does anyone have any examples, tutorials, or experience they could share with me to help me learn how to do this?</p>