How to train/upscale a very large dataset using scikit-learn?
<p>I'm using the sentiment140 dataset which consists of 1.6M tweets to train and analyze the accuracies of different classifiers in the scikit-learn library in python. I am using the following code snippet to vectorize the tweets into feature vectors before feeding them to the classifiers.</p>

<pre><code>vectorizer = CountVectorizer(max_features = 2000)
train_set = (vectorizer.fit_transform(trainX)).toarray()
</code></pre>

<p>After this I train my <code>classifier</code> object which includes <code>GaussianNB()</code>, <code>MultinomialNB()</code>, <code>BernoulliNB()</code>, <code>LogisticRegression()</code>, <code>LinearSVC()</code> and <code>RandomForestClassifier()</code> using the following code snippet:</p>

<pre><code>classifier.fit(train_vectors,trainy)
</code></pre>

<p>However, on converting my vectorizer's transformed set to a numpy array with the <code>toarray()</code> function for the <code>trainset</code>, I see that the program is using up a great deal of memory (about 4-5 GB) for just 100k examples with a feature vector of size 2000 for each example, i.e., a 100,000x2000 feature vector.</p>

<p>This is the maximum my system can do since I only have 8GB of RAM. Can someone give suggestions as to how I may proceed so as to be able to train the entire 1.6M of the training dataset with the available memory by possibly modifying the code anyhow. If I try to do with the above code it would require around 72 gigs of RAM which is not feasible. </p>

<p>I also understand there are provisions to incrementally train certain classifiers with a fraction of the train set iteratively. Classifiers such as <code>MultinomialNB()</code> and <code>BernoulliNB()</code> have provisions for that (using <code>partial_fit</code>) but the other classifiers that I'm also using don't, so that's not an option.</p>