{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/12819","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/12819/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/12819/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/12819/events","html_url":"https://github.com/elastic/elasticsearch/issues/12819","id":100471485,"number":12819,"title":"Are these ES bugs or just wrong implementations by me?","user":{"login":"apanimesh061","id":3243029,"avatar_url":"https://avatars.githubusercontent.com/u/3243029?v=3","gravatar_id":"","url":"https://api.github.com/users/apanimesh061","html_url":"https://github.com/apanimesh061","followers_url":"https://api.github.com/users/apanimesh061/followers","following_url":"https://api.github.com/users/apanimesh061/following{/other_user}","gists_url":"https://api.github.com/users/apanimesh061/gists{/gist_id}","starred_url":"https://api.github.com/users/apanimesh061/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apanimesh061/subscriptions","organizations_url":"https://api.github.com/users/apanimesh061/orgs","repos_url":"https://api.github.com/users/apanimesh061/repos","events_url":"https://api.github.com/users/apanimesh061/events{/privacy}","received_events_url":"https://api.github.com/users/apanimesh061/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-08-12T05:39:40Z","updated_at":"2015-08-12T09:58:32Z","closed_at":null,"body":"I have set the maximum shingle size as 3 for my index. When I am done with indexing I find that the shingles : \r\n1. `yield`\r\n2. `_ yield`\r\n3. `_ _ yield`\r\n\r\nhave the same start and ending offsets. So, if I would need the original sub-string I will always get `yield`!! Why is that and can it be corrected?\r\n\r\nAnother thing that I noticed is that the offsets of the shingles like:\r\n1. `_ yield _`\r\n2. `yield _`\r\n3. `yield _ _`\r\nhave length such that the string has an extra space that the end. e.g. for `_ yield _` if there is a string `\"not yield until\"` then the offsets, is used on the original text, return `\"yield until \"` (notice trailing space). I actually would want `\"not yield until\"`.\r\n\r\nFinally, is there a way we can avoid stop words while creating shingles? In my application the stop words don't play any role.\r\n\r\nFollowing is the settings that I am using for the index:\r\n\r\n    {\r\n       \"test_index\": {\r\n          \"settings\": {\r\n             \"index\": {\r\n                \"refresh_interval\": \"60s\",\r\n                \"number_of_shards\": \"5\",\r\n                \"store\": {\r\n                   \"type\": \"default\"\r\n                },\r\n                \"creation_date\": \"1439324341816\",\r\n                \"analysis\": {\r\n                   \"filter\": {\r\n                      \"snowball_stop_words_en_EN\": {\r\n                         \"type\": \"stop\",\r\n                         \"stopwords_path\": \"snowball.stop\"\r\n                      },\r\n                      \"smart_stop_words_en_EN\": {\r\n                         \"type\": \"stop\",\r\n                         \"stopwords_path\": \"smart.stop\"\r\n                      },\r\n                      \"porter_stemmer_en_EN\": {\r\n                         \"name\": \"porter\",\r\n                         \"type\": \"stemmer\"\r\n                      },\r\n                      \"word_delimiter_en_EN\": {\r\n                         \"type\": \"word_delimiter\",\r\n                         \"stem_english_possessive\": \"true\"\r\n                      },\r\n                      \"default_stop_name_en_EN\": {\r\n                         \"name\": \"_english_\",\r\n                         \"type\": \"stop\"\r\n                      },\r\n                      \"preserve_original_en_EN\": {\r\n                         \"type\": \"word_delimiter\",\r\n                         \"preserve_original\": \"true\"\r\n                      },\r\n                      \"apos_replace_en_EN\": {\r\n                         \"pattern\": \".*\\\\'$\",\r\n                         \"type\": \"pattern_replace\",\r\n                         \"replacement\": \"\"\r\n                      },\r\n                      \"shingle_filter_en_EN\": {\r\n                         \"max_shingle_size\": \"3\",\r\n                         \"min_shingle_size\": \"2\",\r\n                         \"type\": \"shingle\",\r\n                         \"output_unigrams\": \"true\"\r\n                      }\r\n                   },\r\n                   \"analyzer\": {\r\n                      \"test_analyzer\": {\r\n                         \"filter\": [\r\n                            \"lowercase\",\r\n                            \"smart_stop_words_en_EN\",\r\n                            \"preserve_original_en_EN\",\r\n                            \"porter_stemmer_en_EN\",\r\n                            \"asciifolding\",\r\n                            \"apos_replace_en_EN\",\r\n                            \"shingle_filter_en_EN\"\r\n                         ],\r\n                         \"type\": \"custom\",\r\n                         \"tokenizer\": \"standard\"\r\n                      }\r\n                   }\r\n                },\r\n                \"number_of_replicas\": \"1\",\r\n                \"version\": {\r\n                   \"created\": \"1060099\"\r\n                },\r\n                \"uuid\": \"urrFTCEoThyuPjjBYCBgYQ\"\r\n             }\r\n          }\r\n       }\r\n    }\r\n\r\nI use the `test_analyzer` on the documents while indexing.\r\n\r\nI am using ElasticSearch 1.6.0.\r\n\r\nAs an example I have even attached [link](https://www.dropbox.com/s/dipb0quha3wke1f/fft.txt?dl=0) to a text file that I was using.\r\n\r\nThanks.\r\nAny suggestions are welcome."},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/130243525","html_url":"https://github.com/elastic/elasticsearch/issues/12819#issuecomment-130243525","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/12819","id":130243525,"user":{"login":"clintongormley","id":56599,"avatar_url":"https://avatars.githubusercontent.com/u/56599?v=3","gravatar_id":"","url":"https://api.github.com/users/clintongormley","html_url":"https://github.com/clintongormley","followers_url":"https://api.github.com/users/clintongormley/followers","following_url":"https://api.github.com/users/clintongormley/following{/other_user}","gists_url":"https://api.github.com/users/clintongormley/gists{/gist_id}","starred_url":"https://api.github.com/users/clintongormley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/clintongormley/subscriptions","organizations_url":"https://api.github.com/users/clintongormley/orgs","repos_url":"https://api.github.com/users/clintongormley/repos","events_url":"https://api.github.com/users/clintongormley/events{/privacy}","received_events_url":"https://api.github.com/users/clintongormley/received_events","type":"User","site_admin":false},"created_at":"2015-08-12T09:58:31Z","updated_at":"2015-08-12T09:58:31Z","body":"@colings86 actually i think this is worth discussing here.  there may be things we can improve.  Reopening"}}