Decreasing memory consumption in R -- pass by reference / data.table
<p>I've already achieved a substantial speed up (~6.5x) by moving subsetting operations from base <code>data.frame</code> operations to <code>data.table</code> operations. But I'm wondering if I can get any improvement in memory. </p>

<p>My understanding is that R does <strong>not</strong> natively pass-by-reference (eg. <a href="https://stackoverflow.com/questions/2603184/r-pass-by-reference">see here</a>). So, I'm seeking a method (short of re-writing a complex function in <code>Rcpp</code>) to do so. <code>data.table</code> provides some improvement [<strong>after editing my question to include typo caught by @joshua ulrich below</strong>]. But I'm looking for a larger improvement if possible.</p>

<ul>
<li>Another option is possibly the <a href="https://cran.r-project.org/web/packages/R.oo/R.oo.pdf" rel="nofollow noreferrer">R.oo package</a>, though I haven't yet found a good tutorial. (<em>I still need to read <a href="http://www.ci.tuwien.ac.at/Conferences/DSC-2003/Proceedings/Bengtsson.pdf" rel="nofollow noreferrer">this</a>.</em></li>
<li>Would <a href="http://stat.ethz.ch/R-manual/R-devel/library/methods/html/refClass.html" rel="nofollow noreferrer">reference classes</a> help at all?</li>
</ul>

<p><em>In my actual use case, I'm doing simulation in parallel of numerous datasets with optimization via simulated annealing. I'd rather not re-write both simulated annealing and my loss function calculations in Rcpp due to the increased dev time and increased <a href="https://en.wikipedia.org/wiki/Technical_debt" rel="nofollow noreferrer">technical debt</a>.</em></p>

<h3>Example of problem:</h3>

<p>What I'm largely concerned with is removing some subset of observations from a dataset and adding in another subset of observations. A very simple (nonsensical) example is given here. Is there a way to decrease memory usage? My current usage appears to pass-by-value and therefore memory usage (RAM) is roughly doubled. </p>

<pre><code>library(data.table)
set.seed(444L)

df1 &lt;- data.frame(matrix(rnorm(1e7), ncol= 10))
df2 &lt;- data.table(matrix(rnorm(1e7), ncol= 10))

prof_func &lt;- function(df) {
  s1 &lt;- sample(1:nrow(df), size= 500, replace=F)
  s2 &lt;- sample(1:nrow(df), size= 500, replace=F)
  return(rbind(df[-s1,], df[s2,]))
}

dt_m &lt;- df_m &lt;- vector("numeric", length= 500L)

for (i in 1:500) {

  Rprof("./DF_mem.out", memory.profiling = TRUE)
  y &lt;- prof_func(df1)
  Rprof(NULL)
  df &lt;- summaryRprof("./DF_mem.out", memory= "both")
  df_m[i] &lt;- df$by.self$mem.total[which(rownames(df$by.self) == "\"rbind\"")]


  Rprof("./DT_mem.out", memory.profiling = TRUE)
  y2 &lt;- prof_func(df2)
  Rprof(NULL)
  dt &lt;- summaryRprof("./DT_mem.out", memory = "both")
  dt_m[i] &lt;- dt$by.self$mem.total[which(rownames(dt$by.self) == "\"rbind\"")]

}
pryr::object_size(df1)
80 MB
pryr::object_size(df2)
80 MB

# EDITED: via typo / fix from @Joshua Ulrich.
# improvement in memory usage via DT. still not pass-by-reference
quantile(df_m, seq(0,1,.1))
    0%    10%    20%    30%    40%    50%    60%    70%    80%    90%   100% 
379.00 428.60 440.10 447.70 455.36 459.20 466.48 469.89 474.40 482.10 512.60 
quantile(dt_m, seq(0,1,.1))
    0%    10%    20%    30%    40%    50%    60%    70%    80%    90%   100% 
 76.80  84.50  84.50  92.10  92.10  92.10  92.10 107.30 116.46 130.20 157.00 
</code></pre>

<h3>Appendix:</h3>

<pre><code>### speed improvement:
#-----------------------------------------------
library(data.table)
library(microbenchmark)

set.seed(444L)

df1 &lt;- data.frame(matrix(rnorm(1e7), ncol= 10))
df2 &lt;- data.table(matrix(rnorm(1e7), ncol= 10))

microbenchmark(
  df= {
    s1 &lt;- sample(1:nrow(df1), size= 500, replace=F)
    s2 &lt;- sample(1:nrow(df1), size= 500, replace=F)
    df1 &lt;- rbind(df1[-s1,], df1[s2,])
  },
  dt= {
    s1 &lt;- sample(1:nrow(df2), size= 500, replace=F)
    s2 &lt;- sample(1:nrow(df2), size= 500, replace=F)
    df2 &lt;- rbind(df2[-s1,], df2[s2,])

  }, times= 100L)

Unit: milliseconds
 expr      min        lq     mean   median       uq      max neval cld
   df 672.5106 757.65188 814.1582 809.6346 864.6668 998.2290   100   b
   dt  68.1254  85.73178 139.1256 120.3613 148.8243 397.7359   100  a 
</code></pre>