python stanford posttager, java command failed after running for some time
<p>I am using stanford posttager toolkit to tag list of words from academic papers. Here is my codes of this part:</p>

<pre><code>  st = StanfordPOSTagger(stanford_tagger_path, stanford_jar_path, encoding = 'utf8', java_options = '-mx2048m')
  word_tuples = st.tag(document)
</code></pre>

<p>document is a list of words derived from nltk.word_tokenize, they come from mormal academic papers so usually there are several thousand of words (mostly 3000 - 4000). I need to process over 10000 files so I keep calling these functions. My program words fine on a small test set with 270 files, but when the number of file gets bigger, the program gives out this error (Java heap space 2G):</p>

<pre><code> raise OSError('Java command failed : ' + str(cmd))
 OSError: Java command failed
</code></pre>

<p>Note that this error does not occur immediately after the execution, it happens after some time of running. I really don't know the reason. Is this because my 3000 - 4000 words are too much ? Thank you very much for help !(Sorry for the bad edition, the error information is too long) </p>