Can I use TfidfVectorizer in scikit-learn for non-English language? Also how do I read a non-English text in Python?
<p>I have to read a text document which contains both English and non-English (Malayalam specifically) languages in Python. The following I see:</p>

<pre><code>&gt;&gt;&gt;text_english = 'Today is a good day'
&gt;&gt;&gt;text_non_english = 'ആരാണു സന്തോഷമാഗ്രഹിക്കാത്തത'
</code></pre>

<p>Now, if I write a code to extract the first letter using</p>

<pre><code>&gt;&gt;&gt;print(text_english[0])
'T'
</code></pre>

<p>and when I run</p>

<pre><code>&gt;&gt;&gt;print(text_non_english[0])
�
</code></pre>

<p>To get the first letter, I have to write the following</p>

<pre><code>&gt;&gt;&gt;print(text_non_english[0:3])
ആ
</code></pre>

<p>Why this happens?
My aim to extract the words in the text so that I can input it to the tfidf transformer. When I create the tfidf vocabulary from the Malayalam language, there are words which are two letters which is not correct. Actually they are part of the full words. What should i do so that the tfidf transformer takes the full Malayalam word for the transformation instead of taking two letters.</p>

<p>I used the following code for this</p>

<pre><code>&gt;&gt;&gt;useful_text_1[1:3] # contains both English and Malayalam text

&gt;&gt;&gt;vectorizer = TfidfVectorizer(sublinear_tf=True,max_df=0.5,stop_words='english')

# Learn vocabulary and idf, return term-document matrix
&gt;&gt;&gt;vect_2 = vectorizer.fit_transform(useful_text_1[1:3])
&gt;&gt;&gt;vectorizer.vocabulary_
</code></pre>

<p>Some of the words in the vocabulary are as below:</p>

<pre><code>ഷമ
സന
സഹ
ർക
ർത
</code></pre>

<p>The vocabulary is not correct. It is not considering the whole word. How to rectify this?</p>