Create dfm step by step with quanteda
<p>I want to analyze a big (n=500,000) corpus of documents. I am using <code>quanteda</code> in the expectation that  <a href="https://stackoverflow.com/questions/25330753/more-efficient-means-of-creating-a-corpus-and-dtm-with-4m-rows">will be faster</a> than <code>tm_map()</code> from <code>tm</code>. I want to proceed step by step instead of using the automated way with <code>dfm()</code>. I have reasons for this: in one case, I don't want to tokenize before removing stopwords as this would result in many useless bigrams, in another I have to preprocess the text with language-specific procedures.</p>

<p>I would like this sequence to be implemented:<br>
1) remove the punctuation and numbers<br>
2) remove stopwords (i.e. before the tokenization to avoid useless tokens)<br>
3) tokenize using unigrams and bigrams<br>
4) create the dfm</p>

<p>My attempt:</p>

<pre><code>&gt; library(quanteda)
&gt; packageVersion("quanteda")
[1] ‘0.9.8’
&gt; text &lt;- ie2010Corpus$documents$texts
&gt; text.corpus &lt;- quanteda:::corpus(text, docnames=rownames(ie2010Corpus$documents))

&gt; class(text.corpus)
[1] "corpus" "list"

&gt; stopw &lt;- c("a","the", "all", "some")
&gt; TextNoStop &lt;- removeFeatures(text.corpus, features = stopw)
# Error in UseMethod("selectFeatures") : 
# no applicable method for 'selectFeatures' applied to an object of class "c('corpus', 'list')"

# This is how I would theoretically continue: 
&gt; token &lt;- tokenize(TextNoStop, removePunct=TRUE, removeNumbers=TRUE)
&gt; token2 &lt;- ngrams(token,c(1,2))
</code></pre>

<p><strong>Bonus question</strong>
How do I remove sparse tokens in <code>quanteda</code>? (i.e. equivalent of <code>removeSparseTerms()</code> in <code>tm</code>.</p>

<hr>

<p><strong>UPDATE</strong>
At the light of @Ken's answer, here is the code to proceed step by step with <code>quanteda</code>:</p>

<pre><code>library(quanteda)
packageVersion("quanteda")
[1] ‘0.9.8’
</code></pre>

<p>1) Remove custom punctuation and numbers. E.g. notice that the "\n" in the ie2010 corpus</p>

<pre><code>text.corpus &lt;- ie2010Corpus
texts(text.corpus)[1]      # Use texts() to extrapolate text
# 2010_BUDGET_01_Brian_Lenihan_FF
# "When I presented the supplementary budget to this House last April, I said we
# could work our way through this period of severe economic distress. Today, I
# can report that notwithstanding the difficulties of the past eight months, we
# are now on the road to economic recovery.\nIt is

texts(text.corpus)[1] &lt;- gsub("\\s"," ",text.corpus[1])    # remove all spaces (incl \n, \t, \r...)
texts(text.corpus)[1]
2010_BUDGET_01_Brian_Lenihan_FF
# "When I presented the supplementary budget to this House last April, I said we
# could work our way through this period of severe economic distress. Today, I
# can report that notwithstanding the difficulties of the past eight months, we
# are now on the road to economic recovery. It is of e
</code></pre>

<p>A further note on the reason why one may prefer to preprocess. My present corpus is in Italian, a language that has articles connected to the words with an apostrophe. Thus, the straight <code>dfm()</code> can lead to inexact tokenization.
e.g.: </p>

<pre><code>broken.tokens &lt;- dfm(corpus(c("L'abile presidente Renzi. Un'abile mossa di Berlusconi"), removePunct=TRUE))
</code></pre>

<p>will produce two separated tokens for the same word ("un'abile" and "l'abile"), hence the need of an additional step with <code>gsub()</code> here.</p>

<p>2) In <code>quanteda</code> it is not possible to remove stopwords directly in the text before the tokenization. In my previous example "l" and "un" have to be removed not to produce misleading bigrams. This can be handled in <code>tm</code> with <code>tm_map(..., removeWords)</code>. </p>

<p>3) Tokenization </p>

<pre><code>token &lt;- tokenize(text.corpus[1], removePunct=TRUE, removeNumbers=TRUE, ngrams = 1:2)
</code></pre>

<p>4) Create the dfm:</p>

<pre><code>dfm &lt;- dfm(token)
</code></pre>

<p>5) Remove sparse features</p>

<pre><code>dfm &lt;- trim(dfm, minCount = 5)
</code></pre>