Deduplication in nutch 1.11 for sites having same content and different URI
<p>I am using nutch to crawl over intranet sites. I came across a scenario in which 2 sites have same content (compared the view source of both sites using notepad++) but different URI.
e.g.</p>

<pre><code>http://site_name.domain_name.com/a/b/c/index.html
http://site_name.domain_name.com/x/y/z/index.html
</code></pre>

<p>Nutch is indexing either of them but not the both. </p>

<p>How to alter this behavior of nutch and index both of the URLs ?</p>