{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/14538","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14538/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14538/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14538/events","html_url":"https://github.com/elastic/elasticsearch/issues/14538","id":115171560,"number":14538,"title":"Highlighter causing overflow of boolean max clause count","user":{"login":"ppf2","id":7216393,"avatar_url":"https://avatars.githubusercontent.com/u/7216393?v=3","gravatar_id":"","url":"https://api.github.com/users/ppf2","html_url":"https://github.com/ppf2","followers_url":"https://api.github.com/users/ppf2/followers","following_url":"https://api.github.com/users/ppf2/following{/other_user}","gists_url":"https://api.github.com/users/ppf2/gists{/gist_id}","starred_url":"https://api.github.com/users/ppf2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ppf2/subscriptions","organizations_url":"https://api.github.com/users/ppf2/orgs","repos_url":"https://api.github.com/users/ppf2/repos","events_url":"https://api.github.com/users/ppf2/events{/privacy}","received_events_url":"https://api.github.com/users/ppf2/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Highlighting","name":":Highlighting","color":"009800"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-11-04T23:15:04Z","updated_at":"2015-11-05T17:17:22Z","closed_at":null,"body":"The following appears to be the case where the wildcard query (in this case, entered by a Kibana user) caused enough Lucene rewrites so that it overflows the boolean max clause count of 1024 by default.  While we could potentially workaround this by increasing the max clause count in Lucene/ES, it is not the ideal workaround given the memory implications.    \r\n\r\nThis appears to be a highlighter issue based on the stack.  Not sure if the highlighter honors these rewrite settings (https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-multi-term-rewrite.html) ... Is there any safeguard that can be configured to prevent the highlighter from causing the great number of bool queries that is being generated?\r\n\r\n```\r\n[indices:data/read/search[phase/fetch/id]] \r\nCaused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: [logstash-2015.11.03][9]: query[filtered(+host:* +(message:20151103) +(message:main) +(message:dialer))->BooleanFilter(+cache(@timestamp:[1446555600000 TO 1446574259999]))],from[0],size[1000],sort[<custom:\"@timestamp\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@69646af6>!]: Fetch Failed [Failed to highlight field [message]] \r\nat org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:134) \r\nat org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:133) \r\nat org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:194) \r\nat org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:516) \r\nat org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868) \r\nat org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862) \r\nat org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279) \r\nat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36) \r\nat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) \r\nat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) \r\nat java.lang.Thread.run(Thread.java:745) \r\nCaused by: org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to 1024\r\nat org.apache.lucene.search.ScoringRewrite$1.checkMaxClauseCount(ScoringRewrite.java:72) \r\nat org.apache.lucene.search.ScoringRewrite$ParallelArraysTermCollector.collect(ScoringRewrite.java:149)\r\nat org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:79) \r\nat org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:105) \r\nat org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288) \r\nat org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217) \r\nat org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99) \r\nat org.elasticsearch.search.highlight.CustomQueryScorer$CustomWeightedSpanTermExtractor.extractUnknownQuery(CustomQueryScorer.java:89) \r\nat org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:224) \r\nat org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474) \r\nat org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217) \r\nat org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186) \r\nat org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197) \r\nat org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:120) \r\n... 10 more\r\n```"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/154126896","html_url":"https://github.com/elastic/elasticsearch/issues/14538#issuecomment-154126896","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14538","id":154126896,"user":{"login":"nik9000","id":215970,"avatar_url":"https://avatars.githubusercontent.com/u/215970?v=3","gravatar_id":"","url":"https://api.github.com/users/nik9000","html_url":"https://github.com/nik9000","followers_url":"https://api.github.com/users/nik9000/followers","following_url":"https://api.github.com/users/nik9000/following{/other_user}","gists_url":"https://api.github.com/users/nik9000/gists{/gist_id}","starred_url":"https://api.github.com/users/nik9000/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nik9000/subscriptions","organizations_url":"https://api.github.com/users/nik9000/orgs","repos_url":"https://api.github.com/users/nik9000/repos","events_url":"https://api.github.com/users/nik9000/events{/privacy}","received_events_url":"https://api.github.com/users/nik9000/received_events","type":"User","site_admin":false},"created_at":"2015-11-05T17:17:22Z","updated_at":"2015-11-05T17:17:22Z","body":"> Is there any safeguard that can be configured to prevent the highlighter from causing the great number of bool queries that is being generated?\r\n\r\nI honestly don't know if the highlighter respects the rewrite configuration - without trying it I'm 60% sure it does but that isn't saying much. For what its worth the experimental highlighter doesn't have to rewrite most wildcard queries so it won't hit this particular problem."}}