Azure storage incremental copy by modified date
<p>I need to copy one storage account into another. I have created a <code>Runbook</code> and schedule it to run daily. This is an incremental copy.</p>

<p>What I am doing is</p>

<ol>
<li>list the blobs in source storage container </li>
<li>Check the blobs in destination storage container</li>
<li>If it doesn't exist in destination container copy the blob <code>Start-AzureStorageBlobCopy</code></li>
</ol>

<p>While this works for containers with small size, this takes a very long time and is certainly cost ineffective for containers with say 10 million block blobs because every time I run the task I have to go through all those 10 million blobs.</p>

<p>I don't see it in <a href="https://msdn.microsoft.com/en-us/library/dn806392.aspx" rel="nofollow">documentation</a> but is there any way i can use conditional headers like <code>DateModifedSince</code> some thing like <code>Get-AzureStorageBlob -DateModifiedSince date</code> in powershell.</p>

<p>I have not tried but I can see it is possible to use <code>DateModifiedSince</code> in <a href="https://msdn.microsoft.com/en-us/library/dn806392.aspx" rel="nofollow">nodejs library</a></p>

<p>Is there anyway I can do it with powershell so that I can be able to use <code>Runbooks</code>?</p>

<p>EDIT:</p>

<p>Using AzCopy made a copy of storage account that contains 7 million blobs, I uploaded few new blobs and started the azcopy again. It still takes significant amount of time to copy few new uploaded files.</p>

<p><code>AzCopy /Source:$sourceUri /Dest:$destUri /SourceKey:$sourceStorageKey /DestKey:$destStorageAccountKey /S /XO /XN /Y</code></p>

<p>It is possible to filter for a blob with blob name in no time</p>

<p>For example <code>Get-AzureStorageBlob -Blob</code> will return the blob immediately from 7 million records</p>

<p>It should have been possible to filter blob(s) with other properties too..</p>