Merging small files in hadoop
<p>I have a directory (Final Dir) in HDFS in which some files(ex :10 mb) are loading every minute. 
After some time i want to combine all the small files to a large file(ex :100 mb). But the user is continuously pushing files to Final Dir. it is a continuous process.</p>

<p>So for the first time i need to combine the first 10 files to a large file (ex : large.txt) and save file to Finaldir. </p>

<p>Now my question is how i will get the next 10 files excluding the first 10 files?    </p>

<p>can some please help me</p>