"Letter" tokenizer and "word_delimiter" filter not working with underscores
<p>I built an ElasticSearch index using a custom analyzer which uses <code>letter</code> tokenizer and <code>lower_case</code> and <code>word_delimiter</code> token filters. Then I tried searching for documents containing underscore-separated sub-words, e.g. <code>abc_xyz</code>, using only one of the sub-words, e.g. <code>abc</code>, but it didn't come back with any result. When I tried the full-word, i.e. <code>abc_xyz</code>, it did find the document.</p>

<p>Then I changed the document to have dash-separated sub-words instead, e.g. <code>abc-xyz</code> and tried to search by sub-words again and it worked.</p>

<p>To try to understand what is going on, I thought I would check the terms generated for my documents using <code>_termvector</code> service, and the result was identical for both, the underscore-separated sub-words and the dash-separated sub-words, so really I expect the result of searching to be identical in both cases.</p>

<p>Any idea what I could be doing wrong?</p>

<p>If it helps, this is the settings I used for my index:</p>

<pre><code>{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "cmt_value_analyzer": {
            "tokenizer": "letter",
            "filter": [
              "lowercase",
              "my_filter"
            ],
            "type": "custom"
          }
        },
        "filter": {
          "my_filter": {
            "type": "word_delimiter"
          }
        }
      }
    }
  },
  "mappings": {
    "alertmodel": {
      "properties": {
        "name": {
          "analyzer": "cmt_value_analyzer",
          "term_vector": "with_positions_offsets_payloads",
          "type": "string"
        },
        "productId": {
          "type": "double"
        },
        "productName": {
          "analyzer": "cmt_value_analyzer",
          "term_vector": "with_positions_offsets_payloads",
          "type": "string"
        },
        "link": {
          "analyzer": "cmt_value_analyzer",
          "term_vector": "with_positions_offsets_payloads",
          "type": "string"
        },
        "updatedOn": {
          "type": "date"
        }
      }
    }
  }
}
</code></pre>