fast split of data.frame into list of data.frames in R
<p>I have a data.frame with 93 millon elements and 3 numeric variables. The third variable, "component" groups some rows by an id. The data consists of an edge list of a huge graph, the component number indicates the rows that belong to the same connected component.  There are about 83 million such components. </p>

<p>I am now trying t split the data frame into a list 83 million of data.frames. I do this in order to apply some igraph functions to each component. </p>

<p><a href="https://stackoverflow.com/questions/18527051/split-a-large-dataframe-into-a-list-of-data-frames-based-on-common-value-in-colu">This SO answer</a> indicates that <code>split()</code> is the solution for this. </p>

<pre><code>library(dplyr,data.table,igraph)

# d6b: data.frame with edge A, edge B, component, 93 millon rows, 83 million components, object.size=2,4Gb
d6b &lt;- d6a %&gt;% split(f = d6a$component )
# This takes 7,1 hours to run, and creates a 94.8 Gb object

#Then try to run igraph on each element of the list
d6b %&gt;% lapply(graph_from_data_frame,directed = TRUE) -&gt; g6a
#code above ran for 20 hours without finishing
</code></pre>

<p>Is there a faster way to do this? Is there another structure that does not become so large? </p>

<p>EDIT: based on Gregor's comment bellow I changed the workflow:</p>

<pre><code>#Selecting only the non trivial components 
# removing all 1:n or n:1 (incluind the 70mi 1:1)
d6a %&gt;% group_by(component) %&gt;% 
  mutate(N_edges=n(),
         N_cpf=n_distinct(cpf),
         N_pis=n_distinct(pis)) -&gt; d6b #takes 1h
d6b_dt &lt;- data.table(d6b) # takes 11min
d6b_dtf &lt;- d6b_dt[N_cpf&gt;1 &amp; N_pis&gt;1] # 5s
setkey(d6b_dtf, component) #1s
</code></pre>

<p>Then try to implement the suggestion:</p>

<pre><code>d6b_dtf %&gt;% group_by(component) %&gt;% select(cpf,pis) %&gt;% 
  do(graph_from_data_frame, directed = TRUE) -&gt; g_d6b_dtf
</code></pre>

<p>I get the following error message:</p>

<pre><code>Adding missing grouping variables: `component`
Error: Arguments to do() must either be all named or all unnamed
</code></pre>