NLTK stem txt files stored in local folder
<p>I have a large collection of txt files in my local drive that I would like to tokenise and stem their content. The script bellow works fine if you only have 1 document to process, and most NLTK tutorials online do not explain how one would go about if one needs to loop over txt files in local folder (i.e if you have a corpus of documents). I am sure this is a simple loop problem, but I just can't figure out how to stem my files.  </p>

<pre><code> import nltk
 import string
 from collections import Counter

    def get_tokens():
       with open('/Users/myname/Desktop/text.txt', 'r') as shakes:
        text = shakes.read()
        lowers = text.lower()
        no_punctuation = lowers.translate(None, string.punctuation)
        tokens = nltk.word_tokenize(no_punctuation)
        return tokens

    tokens = get_tokens()
    count = Counter(tokens)
    print count.most_common(10)


    from nltk.corpus import stopwords

    tokens = get_tokens()
    filtered = [w for w in tokens if not w in stopwords.words('english')]
    count = Counter(filtered)
    print count.most_common(100)

    from nltk.stem.porter import *

    def stem_tokens(tokens, stemmer):
        stemmed = []
        for item in tokens:
            stemmed.append(stemmer.stem(item))
        return stemmed

    stemmer = PorterStemmer()
    stemmed = stem_tokens(filtered, stemmer)
    count = Counter(stemmed)
    print count.most_common(100)
</code></pre>