How to efficiently store and merge variables in R?
<p>I have 600 <code>.txt</code> files (in total about 250GB). For each file I compute an additional variable which I would need to merge with another separate (smaller) dataset. I do need all observations in all 600 <code>txt</code> files to compute these variables.</p>

<p>Currently I calculate this variable for each file separately (merging all 600 files is impossible), and save only this variable with the unique ID in a separate <code>txt</code> file. </p>

<p>The problem is that I keep running out of memory. Is there a more efficient way to merge this? Any suggestions are very welcome.</p>

<p>My code is the following (it works perfectly when applied on 1 small file):
I do the following for each of the 600 files:
1. compute extra variables,
2. subset dataset containing only the variable I need and a unique ID to merge,
3. save as <code>txt</code> file.</p>

<pre><code>files &lt;- list.files("path", pattern = "*.TXT")

# Loop over files
for (i in 1:length(files))
{
  data &lt;- read.table(files[i], header = TRUE)

  # Compute extra variables
  data$newvar &lt;- (data$v1 * data$v2)
  data &lt;- ddply(data, .(v3,v4,v5), transform, newvar2 = sum(newvar)) 

  # Subset data
  varstokeep &lt;- c("ID", "newvar2")
  data &lt;- data[varstokeep]

  # Save data
  write.table(data, paste("path[", i, "].txt"), sep = "\t")

  rm(data)
}
</code></pre>