Sqoop & Hadoop - How to join/merge old data and new data imported by Sqoop in lastmodified mode?
<p><strong>Background:</strong></p>

<p>I have a table with the following schema on a SQL server. Updates to existing rows is possible and new rows are also added to this table.</p>

<pre><code>unique_id | user_id | last_login_date       | count
123-111   |   111   | 2016-06-18 19:07:00.0 | 180
124-100   |   100   | 2016-06-02 10:27:00.0 | 50
</code></pre>

<p>I am using Sqoop to add incremental updates in lastmodified mode. My --check-column parameter is the last_login_date column. In my <strong>first run</strong>, I got the above two records into Hadoop - let's call this current data. I noted that the last value (the max value of the the check column from this first import) is 2016-06-18 19:07:00.0.</p>

<p>Assuming there is a change on the SQL server side, I now have the following changes on the SQL server side:</p>

<pre><code>unique_id | user_id | last_login_date       | count
123-111   |   111   | 2016-06-25 20:10:00.0 | 200
124-100   |   100   | 2016-06-02 10:27:00.0 | 50
125-500   |   500   | 2016-06-28 19:54:00.0 | 1
</code></pre>

<p>I have the row 123-111 updated with a more recent last_login_date value and the count column has also been updated. I also have a new row 125-500 added.</p>

<p>On my <strong>second</strong> run, sqoop looks at all columns with a last_login_date column greater than my known last value from the previous import - 2016-06-18 19:07:00.0</p>

<p>This gives me only the <strong>changed data</strong>, i.e. 123-111 and 125-500 records. Let's call this - new data. </p>

<p><strong>Question</strong></p>

<p>How do I do a merge join in Hadoop/Hive using the current data and the new data so that I end up with the updated version of 123-111, 124-100, and the newly added 125-500?  </p>