{"action":"created","issue":{"url":"https://api.github.com/repos/wikimedia/search-highlighter/issues/13","labels_url":"https://api.github.com/repos/wikimedia/search-highlighter/issues/13/labels{/name}","comments_url":"https://api.github.com/repos/wikimedia/search-highlighter/issues/13/comments","events_url":"https://api.github.com/repos/wikimedia/search-highlighter/issues/13/events","html_url":"https://github.com/wikimedia/search-highlighter/issues/13","id":67051688,"number":13,"title":"highlighting too many terms causes highlighter to return nothing","user":{"login":"jeacott","id":311357,"avatar_url":"https://avatars.githubusercontent.com/u/311357?v=3","gravatar_id":"","url":"https://api.github.com/users/jeacott","html_url":"https://github.com/jeacott","followers_url":"https://api.github.com/users/jeacott/followers","following_url":"https://api.github.com/users/jeacott/following{/other_user}","gists_url":"https://api.github.com/users/jeacott/gists{/gist_id}","starred_url":"https://api.github.com/users/jeacott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeacott/subscriptions","organizations_url":"https://api.github.com/users/jeacott/orgs","repos_url":"https://api.github.com/users/jeacott/repos","events_url":"https://api.github.com/users/jeacott/events{/privacy}","received_events_url":"https://api.github.com/users/jeacott/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-04-08T05:23:25Z","updated_at":"2015-04-28T11:37:52Z","closed_at":null,"body":"I'm using the latest release of this plugin 1.4.1 with ES 1.4.2\r\n\r\nin the following query I've manually expanded the fields to overcome problems with the highlighter \r\nand wildcard queries (it seems to arbitrarily decide not to return any highlights depending on the query fields - possibly related to internal expansion and this 'too many terms' issue.) eg if I have in my primary search transcript.en*.p.text, then it may or may not work.\r\n\r\nI've tried it against indexes with \"index_options\":\"offsets\", and with\r\n                        \"term_vector\" :\"with_positions_offsets\"\r\nbut it doesnt seem to make much difference with this issue.\r\n \r\nmanually expanding the terms in the highlighter results does give a more consistent result, and below I've omitted all but the first entry for the highlighter fields for brevity.\r\n\r\nwith this many terms in the main query OR the highlight_query, I get zero highlighted results from this experimental highlighter (there is no error) where the fvh highlighter returns good results. \r\nIf I reduce the number of terms I get results from this highlighter.\r\n\r\nthe issue only seems to affect queries where phrase_prefix is used, but it doesn't occur in every query with a phrase_prefix in it. \r\nWhen it occurs for a given query (irrespective of the query terms) it consistently fails.\r\n\r\nif I have a phrase_prefix query ONLY as a highlight_query, and nowhere in the main query then it seems to give results, but if I have it in both it doesn't.\r\n\r\nneither the fvh nor this highlighter seem to respect the common terms cutoff frequency hence my use below of the highlight_query\r\n\r\nI realise this is a bit vague, but I cant pin down exactly what seems to be going on. I love this highlighter mainly due to the fetch_fields feature which improves my performance hugely so I'm very keen to have/help have the issue resolved.\r\n\r\nany hints/help greatly appreciated.\r\n\r\n```\r\n{\r\n  \"from\" : 0,\r\n  \"size\" : 10,\r\n  \"query\" : {\r\n    \"filtered\" : {\r\n      \"query\" : {\r\n        \"bool\" : {\r\n          \"must\" : {\r\n            \"multi_match\" : {\r\n              \"query\" : \"to influence\",\r\n              \"fields\" : [ \"transcript.en.p.text\", \"transcript.en-AU.p.text\",  \"transcript.en-BZ.p.text\",  \"transcript.en-CA.p.text\",  \"transcript.en-CB.p.text\",  \"transcript.en-GB.p.text\",  \"transcript.en-IE.p.text\", \"transcript.en-JM.p.text\", \"transcript.en-NZ.p.text\",  \"transcript.en-PH.p.text\", \"transcript.en-TT.p.text\",  \"transcript.en-US.p.text\", \"transcript.en-ZA.p.text\", \"transcript.en-ZW.p.text\" ],\r\n              \"type\" : \"phrase_prefix\",\r\n              \"slop\" : 2,\r\n              \"minimum_should_match\" : \"75%\",\r\n              \"cutoff_frequency\" : 0.01\r\n            }\r\n          },\r\n          \"should\" : {\r\n            \"multi_match\" : {\r\n              \"query\" : \"to influence\",\r\n              \"fields\" : [\"title^0.2\", \"description^0.7\" ],\r\n              \"type\" : \"best_fields\",\r\n               \"cutoff_frequency\" : 0.01\r\n            }\r\n          }\r\n        }\r\n      },\r\n    }\r\n  },\r\n  \"min_score\" : 1.8E-5,\r\n  \"highlight\" : {\r\n    \"order\" : \"score\",\r\n    \"type\" : \"experimental\",\r\n    \"highlight_query\" : {\r\n      \"multi_match\" : {\r\n        \"query\" : \"to influence\",\r\n        \"fields\" : [\"transcript.en.p.text\", \"transcript.en-AU.p.text\",  \"transcript.en-BZ.p.text\",  \"transcript.en-CA.p.text\",  \"transcript.en-CB.p.text\",  \"transcript.en-GB.p.text\",  \"transcript.en-IE.p.text\", \"transcript.en-JM.p.text\", \"transcript.en-NZ.p.text\",  \"transcript.en-PH.p.text\", \"transcript.en-TT.p.text\",  \"transcript.en-US.p.text\", \"transcript.en-ZA.p.text\", \"transcript.en-ZW.p.text\"],\r\n        \"type\" : \"phrase_prefix\",\r\n        \"slop\" : 2\r\n      }\r\n    },\r\n    \"fields\" : {\r\n      \"transcript.en.p.text\" : {\r\n        \"fragment_size\" : 150,\r\n        \"number_of_fragments\" : 10,\r\n        \"fragmenter\" : \"scan\",\r\n        \"options\" : {\r\n          \"hit_source\" : \"postings\",\r\n          \"skip_if_last_matched\" : false\r\n        }\r\n      }\r\n... more follow \r\n    }\r\n  }\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/wikimedia/search-highlighter/issues/comments/97028761","html_url":"https://github.com/wikimedia/search-highlighter/issues/13#issuecomment-97028761","issue_url":"https://api.github.com/repos/wikimedia/search-highlighter/issues/13","id":97028761,"user":{"login":"nik9000","id":215970,"avatar_url":"https://avatars.githubusercontent.com/u/215970?v=3","gravatar_id":"","url":"https://api.github.com/users/nik9000","html_url":"https://github.com/nik9000","followers_url":"https://api.github.com/users/nik9000/followers","following_url":"https://api.github.com/users/nik9000/following{/other_user}","gists_url":"https://api.github.com/users/nik9000/gists{/gist_id}","starred_url":"https://api.github.com/users/nik9000/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nik9000/subscriptions","organizations_url":"https://api.github.com/users/nik9000/orgs","repos_url":"https://api.github.com/users/nik9000/repos","events_url":"https://api.github.com/users/nik9000/events{/privacy}","received_events_url":"https://api.github.com/users/nik9000/received_events","type":"User","site_admin":false},"created_at":"2015-04-28T11:37:52Z","updated_at":"2015-04-28T11:37:52Z","body":"@jeacott I wouldn't be surprised about FunctionScoreQuery and FiltersFunctionScoreQuery - as you've probably seen they all have to be manually accounted for. So the ones I'm likely to have missed some of the ones I don't use frequently.\r\n\r\nSorry I hadn't seen this 20 days ago - I thought I was subscribed to the project but I'm not getting consistent emails from it.\r\n\r\nIf you expect to put together a pull request I'll certainly review and merge. I expect to make a release sometime in the next few days for https://phabricator.wikimedia.org/T95021."}}