Nutch does not Index on Elasticsearch correctly using Mongodb
<p>I am running Nutch 2.3.1, Mongodb 3.2.9, and Elasticsearch 2.4.1. I have followed a mix of this tutorial: </p>

<p><a href="https://qbox.io/blog/scraping-the-web-with-nutch-for-elasticsearch" rel="nofollow">https://qbox.io/blog/scraping-the-web-with-nutch-for-elasticsearch</a></p>

<p>and this tutorial:</p>

<p><a href="http://www.aossama.com/search-engine-with-apache-nutch-mongodb-and-elasticsearch/" rel="nofollow">http://www.aossama.com/search-engine-with-apache-nutch-mongodb-and-elasticsearch/</a></p>

<p>In order to create a web crawling tool using those aforementioned 3 pieces of software. </p>

<p>Everything works great until it comes down to indexing... as soon as I use the index command from nutch:</p>

<pre><code># bin/nutch index elasticsearch -all
</code></pre>

<p>this happens:</p>

<pre><code>IndexingJob: starting
Active IndexWriters :
ElasticIndexWriter
        elastic.cluster : elastic prefix cluster
        elastic.host : hostname
        elastic.port : port (default 9300)
        elastic.index : elastic index command
        elastic.max.bulk.docs : ealstic bulk index doc counts. (default 250)
        elastic.max.bulk.size : elastic bulk index length. (default 2500500 ~2.5MB)

IndexingJob: done.
</code></pre>

<p>My nutch-site.xml:</p>

<pre><code>&lt;configuration&gt;
  &lt;property&gt;
    &lt;name&gt;storage.data.store.class&lt;/name&gt;
    &lt;value&gt;org.apache.gora.mongodb.store.MongoStore&lt;/value&gt;
    &lt;description&gt;Default class for storing data&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;http.agent.name&lt;/name&gt;
    &lt;value&gt;AOssama Crawler&lt;/value&gt;
  &lt;/property&gt;

  &lt;property&gt;
    &lt;name&gt;plugin.includes&lt;/name&gt;
    &lt;value&gt;protocol-(http|httpclient)|urlfilter-regex|index-(basic|more)|query-(basic|site|url|lang)|indexer-elastic|nutch-extensionpoints|parse-(text|html|msexcel|msword|mspowerpoint|pdf)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)|parse-(html|tika|metatags)|index-(basic|anchor|more|metadata)&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;elastic.host&lt;/name&gt;
    &lt;value&gt;localhost&lt;/value&gt;
  &lt;/property&gt;

  &lt;property&gt;
    &lt;name&gt;elastic.cluster&lt;/name&gt;
    &lt;value&gt;aossama&lt;/value&gt;
  &lt;/property&gt;

  &lt;property&gt;
    &lt;name&gt;elastic.index&lt;/name&gt;
    &lt;value&gt;nutch&lt;/value&gt;
  &lt;/property&gt;

  &lt;property&gt;
    &lt;name&gt;parser.character.encoding.default&lt;/name&gt;
    &lt;value&gt;utf-8&lt;/value&gt;
  &lt;/property&gt;

  &lt;property&gt;
    &lt;name&gt;http.content.limit&lt;/name&gt;
    &lt;value&gt;6553600&lt;/value&gt;
  &lt;/property&gt;
&lt;/configuration&gt;
</code></pre>

<p>I also looked into the ElasticIndexWriter.java code and noticed near line 250 the class that calls the ElasticIndexWriter. I'm digging into that further now, but I'm completely lost as to why this isn't working with Mongo. I'm about to give up and try with Hbase as much as I dislike it.</p>

<p>Thanks!</p>

<p>Joe</p>