Serializing/Compiling HMM MODEL in to FILE in Java
<p>I have successfully evaluated the Lingpipe's HMM implementation of POS Tagging with my own POS Corpus (accuracy of more than 90%).</p>

<p>The Ant File for Evaluating POS HMM on My Own POS Corpus is the same as Brown POS Corpus:</p>

<pre><code>&lt;target name="eval-brown"
        depends="compile"&gt;
  &lt;java classname="EvaluatePos"
        fork="true"
        maxMemory="512M"&gt;
    &lt;jvmarg value="-server"/&gt;
    &lt;classpath refid="classpath.standard"/&gt;
    &lt;arg value="1"/&gt;                 &lt;!-- sent eval rate --&gt;
    &lt;arg value="50000"/&gt;            &lt;!-- toks before eval --&gt;
    &lt;arg value="10"/&gt;                &lt;!-- max n-best --&gt;
    &lt;arg value="8"/&gt;                 &lt;!-- n-gram size --&gt;
    &lt;arg value="128"/&gt;               &lt;!-- num characters --&gt;
    &lt;arg value="8.0"/&gt;               &lt;!-- interpolation ratio --&gt;
    &lt;arg value="BrownPosCorpus"/&gt;  &lt;!-- corpus implementation class --&gt;
    &lt;arg value="${data.pos.brown}"/&gt;    &lt;!-- baseline for data --&gt;
    &lt;arg value="true"/&gt;            &lt;!--  smoothe tags --&gt;
  &lt;/java&gt;
&lt;/target&gt;
</code></pre>

<p>The Class for Evaluating the HMM POS Tagger is: EvaluatePos.java is given as follows.</p>

<pre><code> public class EvaluatePos {

    final int mSentEvalRate;
    final int mToksBeforeEval;
    final int mMaxNBest;
    final int mNGram;
    final int mNumChars;
    final double mLambdaFactor;
    final PosCorpus mCorpus;

    final Set&lt;String&gt; mTagSet = new HashSet&lt;String&gt;();
    HmmCharLmEstimator mEstimator;
    TaggerEvaluator&lt;String&gt; mTaggerEvaluator;
    NBestTaggerEvaluator&lt;String&gt; mNBestTaggerEvaluator;
    MarginalTaggerEvaluator&lt;String&gt; mMarginalTaggerEvaluator;

    int mTrainingSentenceCount = 0;
    int mTrainingTokenCount = 0;

    public EvaluatePos(String[] args) throws Exception {
        mSentEvalRate = Integer.valueOf(args[0]);
        mToksBeforeEval = Integer.valueOf(args[1]);
        mMaxNBest = Integer.valueOf(args[2]);
        mNGram = Integer.valueOf(args[3]);
        mNumChars = Integer.valueOf(args[4]);
        mLambdaFactor = Double.valueOf(args[5]);
        String constructorName = args[6];
        File corpusFile = new File(args[7]);
        Object[] consArgs = new Object[] { corpusFile };
        @SuppressWarnings("rawtypes") // req 2 step
        PosCorpus corpus 
            = (PosCorpus) 
            Class
            .forName(constructorName)
            .getConstructor(new Class[] { File.class })
            .newInstance(consArgs);
        mCorpus = corpus;
    }

    void run() throws IOException {
        System.out.println("\nCOMMAND PARAMETERS:");
        System.out.println("  Sent eval rate=" + mSentEvalRate);
        System.out.println("  Toks before eval=" + mToksBeforeEval);
        System.out.println("  Max n-best eval=" + mMaxNBest);
        System.out.println("  Max n-gram=" + mNGram);
        System.out.println("  Num chars=" + mNumChars);
        System.out.println("  Lambda factor=" + mLambdaFactor);

        CorpusProfileHandler profileHandler = new CorpusProfileHandler();
        parseCorpus(profileHandler);
        String[] tags = mTagSet.toArray(Strings.EMPTY_STRING_ARRAY);
        Arrays.sort(tags);
        Set&lt;String&gt; tagSet = new HashSet&lt;String&gt;();
        for (String tag : tags)
            tagSet.add(tag);

        System.out.println("\nCORPUS PROFILE:");
        System.out.println("  Corpus class=" + mCorpus.getClass().getName());
        System.out.println("  #Sentences="
                           + mTrainingSentenceCount);
        System.out.println("  #Tokens=" + mTrainingTokenCount);
        System.out.println("  #Tags=" + tags.length);
        System.out.println("  Tags=" + Arrays.asList(tags));

        System.out.println("\nEVALUATION:");
        mEstimator
            = new HmmCharLmEstimator(mNGram,mNumChars,mLambdaFactor);
        for (int i = 0; i &lt; tags.length; ++i)
            mEstimator.addState(tags[i]);

        HmmDecoder decoder
            = new HmmDecoder(mEstimator); // no caching
        boolean storeTokens = true;
        mTaggerEvaluator
            = new TaggerEvaluator&lt;String&gt;(decoder,storeTokens);
        mNBestTaggerEvaluator
            = new NBestTaggerEvaluator&lt;String&gt;(decoder,mMaxNBest,mMaxNBest);
        mMarginalTaggerEvaluator
            = new MarginalTaggerEvaluator&lt;String&gt;(decoder,tagSet,storeTokens);

        LearningCurveHandler evaluationHandler
            = new LearningCurveHandler();
        parseCorpus(evaluationHandler);

        System.out.println("\n\n\nFINAL REPORT");

        System.out.println("\n\nFirst Best Evaluation");
        System.out.println(mTaggerEvaluator.tokenEval());

        System.out.println("\n\nN Best Evaluation");
        System.out.println(mNBestTaggerEvaluator.nBestHistogram());

    }

    void parseCorpus(ObjectHandler&lt;Tagging&lt;String&gt;&gt; handler) throws IOException {
        Parser&lt;ObjectHandler&lt;Tagging&lt;String&gt;&gt;&gt; parser = mCorpus.parser();
        parser.setHandler(handler);
        Iterator&lt;InputSource&gt; it = mCorpus.sourceIterator();
        while (it.hasNext()) {
            InputSource in = it.next();
            parser.parse(in);
        }
    }

    class CorpusProfileHandler implements ObjectHandler&lt;Tagging&lt;String&gt;&gt; {
        public void handle(Tagging&lt;String&gt; tagging) {
            ++mTrainingSentenceCount;
            mTrainingTokenCount += tagging.size();
            for (int i = 0; i &lt; tagging.size(); ++i)
                mTagSet.add(tagging.tag(i));
        }
    }

    class LearningCurveHandler implements ObjectHandler&lt;Tagging&lt;String&gt;&gt; {
        Set&lt;String&gt; mKnownTokenSet = new HashSet&lt;String&gt;();
        int mUnknownTokensTotal = 0;
        int mUnknownTokensCorrect = 0;
        public void handle(Tagging&lt;String&gt; tagging) {
            if (mEstimator.numTrainingTokens() &gt; mToksBeforeEval
                &amp;&amp; mEstimator.numTrainingCases() % mSentEvalRate == 0) {

                mTaggerEvaluator.handle(tagging);
                mNBestTaggerEvaluator.handle(tagging);
                mMarginalTaggerEvaluator.handle(tagging);
                System.out.println("\nTest Case "
                                   + mTaggerEvaluator.numCases());
                System.out.println("First Best Last Case Report");
                System.out.println(mTaggerEvaluator.lastCaseToString(mKnownTokenSet));
                System.out.println("N-Best Last Case Report");
                System.out.println(mNBestTaggerEvaluator.lastCaseToString(5));
                System.out.println("Marginal Last Case Report");
                System.out.println(mMarginalTaggerEvaluator.lastCaseToString(5));
                System.out.println("Cumulative Evaluation");
                System.out.print("    Estimator:  #Train Cases="
                                 + mEstimator.numTrainingCases());
                System.out.println(" #Train Toks="
                                   + mEstimator.numTrainingTokens());
                ConfusionMatrix tokenEval = mTaggerEvaluator.tokenEval().confusionMatrix();
                System.out.println("    First Best Accuracy (All Tokens) = "
                                   + tokenEval.totalCorrect() 
                                   + "/" + tokenEval.totalCount()
                                   + " = " + tokenEval.totalAccuracy());
                ConfusionMatrix unkTokenEval = mTaggerEvaluator.unknownTokenEval(mKnownTokenSet).confusionMatrix();
                mUnknownTokensTotal += unkTokenEval.totalCount();
                mUnknownTokensCorrect += unkTokenEval.totalCorrect();
                System.out.println("    First Best Accuracy (Unknown Tokens) = "
                                   + mUnknownTokensCorrect
                                   + "/" + mUnknownTokensTotal
                                   + " = " + (mUnknownTokensCorrect/(double)mUnknownTokensTotal));
            }
            // train after eval
            mEstimator.handle(tagging);
            for (int i = 0; i &lt; tagging.size(); ++i)
                mKnownTokenSet.add(tagging.token(i));
        }
    }

    public static void main(String[] args)
        throws Exception {

        new EvaluatePos(args).run();
    }
}
</code></pre>

<p>My question now is that how to create the HMM model file to use it as a feature in Chain CRF based NER.</p>

<p>How was the pos-en-general-brown.HiddenMarkovModel in The Lingpipe ../../models folder created?</p>

<p>I'm using the BrownPosCorpus.java, BrownPosParser.java and EvaluatePos.java</p>

<p>Where shall I put the following code to create the pos hmm model file?</p>

<pre><code>// write output to file
        File modelFile = new File(args[1]);
        AbstractExternalizable.compileTo(estimator,modelFile);
</code></pre>

<p>And what changes can be made to the Ant File so as to create the pos hmm model file?</p>

<p>I want to use the POS HMM MODEL FILE as a feature in the Chain CRF Feature Extractor:</p>

<pre><code>...
    static final File POS_HMM_FILE
        = new File("../../models/pos-en-general-brown.HiddenMarkovModel");
...
</code></pre>

<p>Best regards.</p>