R Text Classification with 800K documents
<p>I have to do some work on text classification that contains 800K texts. I've been trying to run an practical example I found in the following link:</p>

<p><a href="http://garonfolo.dk/herbert/2015/05/r-text-classification-using-a-k-nearest-neighbour-model/" rel="nofollow noreferrer">http://garonfolo.dk/herbert/2015/05/r-text-classification-using-a-k-nearest-neighbour-model/</a></p>

<p>All has been going well until I've got the to the following instruction:</p>

<pre><code># Transform dtm to matrix to data frame - df is easier to work with
mat.df &lt;- as.data.frame(data.matrix(dtm), stringsAsfactors = FALSE)
</code></pre>

<p>After having this run for several hours I've got an error message:</p>

<pre><code>Error: cannot allocate vector of size 583.9 Gb
In addition: Warning messages:
1: In vector(typeof(x$v), prod(nr, nc)) :
  Reached total allocation of 8076Mb: see help(memory.size)
2: In vector(typeof(x$v), prod(nr, nc)) :
  Reached total allocation of 8076Mb: see help(memory.size)
3: In vector(typeof(x$v), prod(nr, nc)) :
  Reached total allocation of 8076Mb: see help(memory.size)
4: In vector(typeof(x$v), prod(nr, nc)) :
  Reached total allocation of 8076Mb: see help(memory.size)
</code></pre>

<p>Is there a way to overcome this error?</p>

<p>Would it be possible for example to split data.matrix(dtm) to run the job in chunks and then merge them somehow? Or is it better to approach this in another way or in Python?</p>

<p>Thanks</p>